Oct 30 21:06:52.624160 ip-10-0-129-84 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:06:52.624178 ip-10-0-129-84 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:06:52.624187 ip-10-0-129-84 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:06:52.624509 ip-10-0-129-84 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:07:02.822845 ip-10-0-129-84 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:07:02.822864 ip-10-0-129-84 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6520064ea9514dcba649e0285e1c2d95 --
Oct 30 21:08:47.901558 ip-10-0-129-84 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:08:48.278263 ip-10-0-129-84 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:08:48.278263 ip-10-0-129-84 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:08:48.278263 ip-10-0-129-84 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:08:48.278263 ip-10-0-129-84 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:08:48.284990 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.284892    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:08:48.284990 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.284983    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:08:48.284990 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.284987    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:08:48.284990 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.284990    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:08:48.284990 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.284993    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:08:48.284990 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.284997    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:08:48.284990 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285000    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285002    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285005    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285009    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285011    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285014    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285017    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285019    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285023    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285025    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285028    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285030    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285033    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285035    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285038    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285042    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285046    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285049    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285052    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:08:48.285241 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285055    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285057    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285059    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285063    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285066    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285068    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285071    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285074    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285076    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285079    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285082    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285084    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285087    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285090    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285092    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285095    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285099    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285101    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285105    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285107    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:08:48.285686 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285110    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285112    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285115    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285118    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285121    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285123    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285126    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285128    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285131    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285133    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285136    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285138    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285141    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285143    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285146    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285148    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285151    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285170    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285174    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285179    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:08:48.286165 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285183    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285186    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285188    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285191    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285193    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285196    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285199    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285202    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285205    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285207    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285210    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285212    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285215    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285218    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285220    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285223    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285225    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285228    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285232    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285235    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:08:48.286631 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285238    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285240    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285243    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285246    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285249    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285251    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285254    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285257    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285259    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285262    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285264    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285267    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285269    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285272    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285275    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285277    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285281    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285283    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285288    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:08:48.287102 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285291    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.285293    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287088    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287097    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287101    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287105    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287108    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287111    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287114    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287116    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287121    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287125    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287129    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287132    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287134    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287137    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287140    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287142    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:08:48.287615 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287145    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287148    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287150    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287169    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287172    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287175    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287177    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287180    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287182    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287186    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287188    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287191    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287193    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287197    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287199    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287202    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287205    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287207    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287210    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287213    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:08:48.288058 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287215    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287218    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287220    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287223    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287225    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287228    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287230    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287234    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287236    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287239    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287242    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287244    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287246    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287250    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287252    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287255    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287257    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287259    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287262    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287264    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:08:48.288546 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287267    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287269    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287272    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287274    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287277    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287279    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287282    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287284    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287287    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287290    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287293    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287295    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287298    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287300    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287303    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287305    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287308    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287310    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287315    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:08:48.289020 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287318    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287322    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287324    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287327    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287330    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287332    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287335    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287338    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287341    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287343    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287352    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287355    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287358    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287360    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287362    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287365    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287368    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287371    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287374    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287377    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:08:48.289483 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287379    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287382    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287384    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287387    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287390    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287392    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287395    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287398    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287400    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287406    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287409    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287907    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287912    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287916    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287918    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287921    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287923    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287926    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287929    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:08:48.289960 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287931    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287934    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287936    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287939    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287942    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287944    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287947    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287950    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287952    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287955    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287957    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287962    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287967    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287970    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287973    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287976    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287978    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287981    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287983    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287986    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:08:48.290428 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287989    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287991    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287994    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.287997    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288000    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288002    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288005    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288014    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288017    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288020    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288022    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288025    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288028    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288030    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288033    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288035    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288038    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288040    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288043    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:08:48.290907 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288045    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288050    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288053    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288057    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288060    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288062    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288065    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288068    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288070    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288073    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288075    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288078    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288080    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288083    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288086    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288089    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288091    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288094    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288096    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:08:48.291376 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288099    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288101    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288104    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288107    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288109    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288112    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288114    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288116    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288119    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288121    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288124    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288126    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288129    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288131    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288134    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288136    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288139    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288142    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288144    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288147    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:08:48.291820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288149    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288167    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288171    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288175    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288178    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288181    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288183    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288186    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288189    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288192    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288194    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288197    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288199    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288201    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288204    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288208    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288210    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288213    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288217    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.288219    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:08:48.292341 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289377    2581 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289388    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289394    2581 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289399    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289404    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289408    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289412    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289416    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289420    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289423    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289427    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289430    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289433    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289436    2581 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289439    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289442    2581 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289445    2581 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289448    2581 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289453    2581 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289456    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289459    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289462    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289466    2581 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289470    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:08:48.292824 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289473    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289477    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289480    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289483    2581 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289486    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289490    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289493    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289496    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289501    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289504    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289508    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289511    2581 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289514    2581 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289517    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289521    2581 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289524    2581 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289527    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289530    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289533    2581 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289543    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289546    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289549    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289552    2581 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289555    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289558    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:08:48.293417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289561    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289563    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289569    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289572    2581 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289574    2581 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289579    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289582    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289585    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289588    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289591    2581 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289594    2581 flags.go:64] FLAG: --help="false"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289597    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289600    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289604    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289606    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289610    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289613    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289616    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289619    2581 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289622    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289625    2581 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289628    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289631    2581 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289634    2581 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:08:48.294023 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289637    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289640    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289643    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289646    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289649    2581 flags.go:64] FLAG: --lock-file=""
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289652    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289655    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289658    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289664    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289667    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289671    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289674    2581 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289677    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289680    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289683    2581 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289686    2581 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289691    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289694    2581 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289698    2581 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289701    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289704    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289711    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289714    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289717    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289720    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:08:48.294695 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289723    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289730    2581 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289735    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289738    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289741    2581 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289744    2581 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289746    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289750    2581 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289752    2581 flags.go:64] FLAG: --port="10250"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289756    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289758    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-013d510f19a92b96b"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289762    2581 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289765    2581 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289767    2581 flags.go:64] FLAG: --register-node="true"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289770    2581 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289774    2581 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289777    2581 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289782    2581 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289785    2581 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289788    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289791    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289795    2581 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289798    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289800    2581 flags.go:64] FLAG: --runonce="false"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289803    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289807    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289810    2581 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:08:48.295310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289813    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289816    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289819    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289822    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289826    2581 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289828    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289831    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289835    2581 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289838    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289841    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289844    2581 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289847    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289852    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289855    2581 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289858    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289863    2581 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289866    2581 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289869    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289872    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289874    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289878    2581 flags.go:64] FLAG: --v="2"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289882    2581 flags.go:64] FLAG: --version="false"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289888    2581 flags.go:64] FLAG: --vmodule=""
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289892    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.289895    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:08:48.295982 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290032    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290036    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290040    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290043    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290045    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290048    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290051    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290053    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290056    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290059    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290062    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290064    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290067    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290070    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290074    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290076    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290079    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290081    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290084    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:08:48.296851 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290088    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290091    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290094    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290097    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290100    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290103    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290105    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290108    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290110    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290113    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290116    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290120    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290124    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290128    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290130    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290133    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290135    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290138    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290141    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290144    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:08:48.297674 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290146    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290149    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290165    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290168    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290171    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290174    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290177    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290181    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290183    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290186    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290189    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290191    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290194    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290197    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290200    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290202    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290205    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290207    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290210    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:08:48.298543 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290213    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290215    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290218    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290221    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290224    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290226    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290229    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290231    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290234    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290237    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290239    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290242    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290244    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290247    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290249    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290252    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290254    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290257    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290260    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290263    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:08:48.299125 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290266    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290269    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290272    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290274    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290277    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290280    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290282    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290285    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290288    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290290    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290292    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290295    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290298    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290300    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290303    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290305    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290308    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290310    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290313    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:08:48.299820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290315    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290318    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290320    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290323    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290325    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290328    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290331    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290333    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290336    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.290363    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290405    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290409    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290413    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290416    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290420    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290423    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:08:48.300744 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290426    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290428    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290431    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290434    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290436    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290439    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290442    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290444    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290447    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290449    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290452    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290454    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290457    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290461    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290464    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290467    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290470    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290472    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290475    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:08:48.301446 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290477    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290481    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290484    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290486    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290489    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290493    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290496    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290500    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290503    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290506    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290509    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290513    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290516    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290519    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290522    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290525    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290527    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290530    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290532    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:08:48.301946 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290535    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290538    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290540    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290543    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290545    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290548    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290550    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290553    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290555    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290558    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290560    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290563    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290566    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290568    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290572    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290574    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290577    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290580    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290582    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:08:48.302611 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290585    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290587    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290590    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290592    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290595    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290598    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290601    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290603    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290606    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290608    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290611    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290614    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290616    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290619    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290621    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290624    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290627    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290629    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290632    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290634    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:08:48.303395 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290637    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290639    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290642    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290644    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290647    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290649    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290652    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290655    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290658    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290660    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290663    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290665    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290668    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290670    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290673    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290676    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290678    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290681    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290684    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:08:48.304015 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290686    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290689    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290692    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.290695    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.298748    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.298768    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.299442    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.299454    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299540    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299549    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299555    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299560    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299565    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299571    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299575    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299579    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299584    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299588    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299592    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:08:48.304725 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299596    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299603    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299610    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299616    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299622    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299626    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299632    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299638    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299645    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299649    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299653    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299658    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299662    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299667    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299672    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299676    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299680    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299685    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299691    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:08:48.305188 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299695    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299700    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299705    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299709    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299713    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299718    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299722    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299726    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299730    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299735    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299739    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299743    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299747    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299751    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299756    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299760    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299764    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299768    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299773    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299777    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:08:48.305701 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299781    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299785    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299789    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299794    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299799    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299804    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299808    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299813    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299817    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299821    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299825    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299830    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299836    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299840    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299844    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299848    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299853    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299858    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299862    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299866    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:08:48.306206 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299870    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299875    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299879    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299883    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299888    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299892    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299896    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299900    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299905    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299909    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299913    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299917    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299922    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299926    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299930    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299934    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299939    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299943    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299947    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299952    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:08:48.306748 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299956    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299960    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299964    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299969    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299973    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299978    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299982    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299987    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299991    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299995    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.299999    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300004    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300008    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300012    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300016    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300020    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:08:48.307237 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.300061    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300289    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300300    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300305    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300310    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300315    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300319    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300323    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300328    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300332    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300337    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300341    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300345    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300349    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300353    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300357    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300362    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300366    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300370    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:08:48.307722 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300375    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300379    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300383    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300389    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300393    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300397    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300401    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300405    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300410    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300414    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300418    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300422    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300426    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300430    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300434    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300438    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300443    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300448    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300456    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:08:48.308616 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300461    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300467    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300474    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300480    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300485    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300489    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300493    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300498    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300502    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300506    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300510    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300515    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300519    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300523    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300527    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300531    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300535    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300540    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300544    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:08:48.309086 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300548    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300553    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300558    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300562    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300566    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300571    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300575    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300579    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300583    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300587    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300591    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300595    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300600    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300604    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300608    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300612    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300617    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300621    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300625    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300630    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:08:48.309555 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300634    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300638    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300642    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300646    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300650    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300654    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300659    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300662    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300667    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300671    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300675    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300680    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300684    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300688    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300692    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300696    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300700    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300704    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300709    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300713    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:08:48.310051 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300717    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300721    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300725    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300729    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300733    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300737    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300741    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300745    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300749    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300754    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.300796    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300847    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300853    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300858    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300862    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300866    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:08:48.310528 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300870    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300874    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300879    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300883    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300887    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300891    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300895    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300899    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300904    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300909    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300913    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300917    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300921    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300926    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300929    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300933    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300937    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300942    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300946    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300950    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:08:48.310944 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300954    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300958    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300962    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300967    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300974    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300978    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300982    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300986    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300991    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300995    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.300999    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301003    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301007    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301011    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301015    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301020    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301024    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301028    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301032    2581 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301037    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:08:48.311788 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301041    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301046    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301050    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301054    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301058    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301062    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301066    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301071    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301075    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301079    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301084    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301088    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301091    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301095    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301099    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301103    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301110    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301114    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301118    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:08:48.312606 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301122    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301127    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301133    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301139    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301144    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301148    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301174    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301182    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301187    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301191    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301195    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301200    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301204    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301208    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301212    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301218    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301222    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301227    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:08:48.313372 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301231    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301235    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301239    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301244    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301248    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301252    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301256    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301261    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301264    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301269    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301273    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301280    2581 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301284    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301288    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301293    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301297    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301301    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301306    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301310    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301315    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:08:48.313938 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301319    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301323    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301327    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:48.301332    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.301413    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.304395    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.305520    2581 server.go:1014] "Starting client certificate rotation"
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.305622    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.305664    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.308194    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:08:48.314438 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.313799    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:08:48.324641 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.324622    2581 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:08:48.328948 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.328929    2581 log.go:25] "Validated CRI v1 image API"
Oct 30 21:08:48.330238 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.330087    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:08:48.332969 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.332947    2581 fs.go:135] Filesystem UUIDs: map[216814b9-f443-4d64-97e0-a5454c6381d8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8678182a-faa6-41bb-a551-4c87234e0796:/dev/nvme0n1p4]
Oct 30 21:08:48.333032 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.332967    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:08:48.338123 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.338018    2581 manager.go:217] Machine: {Timestamp:2026-10-30 21:08:48.336585851 +0000 UTC m=+0.335046001 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096601 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2de583b017580c15ff6a24643a953a SystemUUID:ec2de583-b017-580c-15ff-6a24643a953a BootID:6520064e-a951-4dcb-a649-e0285e1c2d95 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:07:b8:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:07:b8:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:6c:21:da:4a:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:08:48.338214 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.338125    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:08:48.338261 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.338248    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:08:48.341915 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.341892    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:08:48.342053 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.341916    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:08:48.342104 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.342055    2581 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:08:48.342104 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.342063    2581 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:08:48.342167 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.342105    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:08:48.342167 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.342123    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:08:48.343458 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.343446    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:08:48.348185 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.348173    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:08:48.348299 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.348290    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:08:48.349152 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.349143    2581 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:08:48.349193 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.349169    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:08:48.349193 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.349180    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:08:48.349193 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.349190    2581 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:08:48.349284 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.349199    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:08:48.349580 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.349562    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:08:48.352115 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.352094    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:08:48.359479 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.359466    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:08:48.360447 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.360432    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:08:48.360485 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.360472    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:08:48.360706 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.360691    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:08:48.360735 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.360720    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:08:48.360761 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.360735    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:08:48.360761 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.360747    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:08:48.360813 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.360785    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:08:48.360813 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.360800    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:08:48.361396 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.361387    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:08:48.361433 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.361397    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:08:48.361433 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.361404    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:08:48.361433 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.361410    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:08:48.361433 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.361418    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:08:48.361433 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.361427    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:08:48.362181 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.362172    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:08:48.362217 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.362183    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:08:48.366352 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.366338    2581 server.go:1263] "Started kubelet"
Oct 30 21:08:48.366443 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.366399    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:08:48.367962 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.367910    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:08:48.368104 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.368080    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:08:48.368224 ip-10-0-129-84 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:08:48.369639 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.369618    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:08:48.372571 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.372214    2581 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:08:48.374054 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.374028    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:08:48.377342 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.377325    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:08:48.377972 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.377953    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:08:48.378084 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.378062    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:48.378766 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378748    2581 factory.go:55] Registering systemd factory
Oct 30 21:08:48.378766 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378764    2581 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:08:48.378894 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378792    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:08:48.378894 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378794    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:08:48.378894 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378810    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:08:48.378996 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378896    2581 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:08:48.378996 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378907    2581 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:08:48.378996 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378967    2581 factory.go:153] Registering CRI-O factory
Oct 30 21:08:48.378996 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.378977    2581 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:08:48.379178 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.379019    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:08:48.379178 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.379033    2581 factory.go:103] Registering Raw factory
Oct 30 21:08:48.379178 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.379042    2581 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:08:48.380084 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.380069    2581 manager.go:319] Starting recovery of all containers
Oct 30 21:08:48.386968 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.386947    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:08:48.388916 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.388894    2581 manager.go:324] Recovery completed
Oct 30 21:08:48.394285 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.393472    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:08:48.395952 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.394664    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:08:48.396058 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.394664    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:08:48.396207 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.394635    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18e36ad25d66f59a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-10-30 21:08:48.366294426 +0000 UTC m=+0.364754580,LastTimestamp:2026-10-30 21:08:48.366294426 +0000 UTC m=+0.364754580,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Oct 30 21:08:48.397487 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.397464    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:08:48.397575 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.397504    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:08:48.397575 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.397529    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:08:48.399178 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.398550    2581 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:08:48.399178 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.398565    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:08:48.399178 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.398583    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:08:48.399178 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.398586    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:08:48.399417 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.399390    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:08:48.399473 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.399419    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:08:48.399473 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.399432    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:08:48.401461 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.401445    2581 policy_none.go:50] "Start"
Oct 30 21:08:48.401537 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.401465    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:08:48.401537 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.401479    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:08:48.403602 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.403587    2581 policy_none.go:44] "Start"
Oct 30 21:08:48.421757 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.421523    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18e36ad25f42eea0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-10-30 21:08:48.397487776 +0000 UTC m=+0.395947928,LastTimestamp:2026-10-30 21:08:48.397487776 +0000 UTC m=+0.395947928,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Oct 30 21:08:48.422354 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.422305    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:08:48.422354 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.422307    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:08:48.422354 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.422310    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:08:48.424511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.424489    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pp8bj"
Oct 30 21:08:48.431307 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.431284    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pp8bj"
Oct 30 21:08:48.436235 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.436124    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18e36ad25f436536  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-10-30 21:08:48.397518134 +0000 UTC m=+0.395978290,LastTimestamp:2026-10-30 21:08:48.397518134 +0000 UTC m=+0.395978290,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.436925    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.439533    2581 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.439566    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.439578    2581 server.go:87] "Starting device plugin registration server"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.439867    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.439881    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.440039    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.440127    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.440135    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.440643    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.440695    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.453032    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.454240    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.454258    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.454290    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:08:48.460189 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.454319    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:08:48.470051 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.470034    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:08:48.540539 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.540442    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:08:48.542855 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.542833    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:08:48.542975 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.542866    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:08:48.542975 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.542879    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:08:48.542975 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.542903    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.555244 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.555221    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"]
Oct 30 21:08:48.555317 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.555279    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:08:48.556133 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.556114    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:08:48.556232 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.556142    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:08:48.556232 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.556169    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:08:48.558331 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.558318    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:08:48.558490 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.558475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.558528 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.558505    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:08:48.559103 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.559082    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:08:48.559224 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.559113    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:08:48.559224 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.559123    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:08:48.559224 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.559082    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:08:48.559224 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.559202    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:08:48.559224 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.559224    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:08:48.561122 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.561107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.561190 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.561137    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:08:48.561266 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.561246    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.561313 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.561269    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-84.ec2.internal\": node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:48.561858 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.561842    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:08:48.561945 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.561869    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:08:48.561945 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.561883    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:08:48.593679 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.593654    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-84.ec2.internal\" not found" node="ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.595147 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.595130    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:48.596268 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.596252    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-84.ec2.internal\" not found" node="ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.679563 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.679538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.679664 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.679573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.679664 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.679593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd04ddc0cbac70d5dc4161808c80da69-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bd04ddc0cbac70d5dc4161808c80da69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.695400 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.695381    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:48.780194 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.780139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.780272 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.780200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.780272 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.780249    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.780272 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.780257    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.780363 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.780298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd04ddc0cbac70d5dc4161808c80da69-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bd04ddc0cbac70d5dc4161808c80da69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.780363 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.780332    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd04ddc0cbac70d5dc4161808c80da69-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bd04ddc0cbac70d5dc4161808c80da69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.796067 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.796021    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:48.896231 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.896196    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:48.899381 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.899364    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.901492 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:48.901480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:48.997328 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:48.997133    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:49.097760 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.097689    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:49.198686 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.198653    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:08:49.268398 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.268372    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:08:49.279193 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.279172    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:49.303815 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.303798    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:08:49.304979 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.304954    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:08:49.305075 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.305055    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:08:49.305122 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.305107    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:08:49.305969 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.305952    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:08:49.349607 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.349536    2581 apiserver.go:52] "Watching apiserver"
Oct 30 21:08:49.353137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.353121    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:08:49.378135 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.378110    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:08:49.387497 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.387478    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:08:49.387806 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.387787    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw","openshift-cluster-node-tuning-operator/tuned-99v8c","openshift-multus/multus-w5hw9","openshift-network-operator/iptables-alerter-slp8w","openshift-dns/node-resolver-tsz8h","openshift-image-registry/node-ca-q5wbv","openshift-multus/multus-additional-cni-plugins-h2wqw","openshift-multus/network-metrics-daemon-p67fc","openshift-ovn-kubernetes/ovnkube-node-tlfh6","kube-system/global-pull-secret-syncer-kmwrj","kube-system/konnectivity-agent-n9vj8","openshift-network-diagnostics/network-check-target-vts2m"]
Oct 30 21:08:49.390833 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.390815    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.393733 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.393715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:49.396612 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.396591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.399501 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.399482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.402445 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.402423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.404724 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.404700    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:08:49.404724 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.404710    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:08:49.404874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.404857    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jv27h\""
Oct 30 21:08:49.405255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.405240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.408283 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.408252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.411062 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.411046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:49.411152 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.411103    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:08:49.411427 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.411413    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:08:49.411571 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.411559    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:08:49.411702 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.411689    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:08:49.411795 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.411785    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:08:49.411861 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.411792    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2tg5\""
Oct 30 21:08:49.411861 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.411797    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b5899\""
Oct 30 21:08:49.411861 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.411832    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:08:49.412647 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.412630    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:08:49.412722 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.412710    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:08:49.412773 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.412747    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wjvtw\""
Oct 30 21:08:49.414006 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.413987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.415677 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.415662    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nxx4w\""
Oct 30 21:08:49.415931 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.415908    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:08:49.416081 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.415967    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:08:49.416081 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.415978    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:08:49.416081 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.415997    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:08:49.417183 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.417154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:49.417264 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.417229    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:08:49.418495 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.418470    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:08:49.418622 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.418508    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hnk74\""
Oct 30 21:08:49.418622 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.418549    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:08:49.418794 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.418774    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:08:49.420432 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.420408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.422825 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.422773    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s8m2s\""
Oct 30 21:08:49.422936 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.422860    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:08:49.422936 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.422860    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:08:49.422936 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.422888    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.423071 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.422969    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:08:49.426635 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.426612    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:08:49.426726 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.426681    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-26jx5"
Oct 30 21:08:49.434219 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.434191    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-26jx5"
Oct 30 21:08:49.439740 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.439692    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:03:49 +0000 UTC" deadline="2028-05-23 03:16:06.771226208 +0000 UTC"
Oct 30 21:08:49.439740 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.439736    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13686h7m17.33149236s"
Oct 30 21:08:49.480289 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.480268    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:08:49.486356 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-tmp\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.486450 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-system-cni-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.486450 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-kubelet\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.486450 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-ovn\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.486552 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-konnectivity-ca\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:49.486552 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.486552 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnk9\" (UniqueName: \"kubernetes.io/projected/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kube-api-access-tbnk9\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.486552 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.486676 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.486676 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-config\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.486676 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-cni-binary-copy\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.486676 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-multus\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.486839 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-multus-certs\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.486839 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-etc-kubernetes\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.486839 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-agent-certs\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:49.486839 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42109964-a1d9-479b-b626-76b60f9f49da-hosts-file\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.486839 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42109964-a1d9-479b-b626-76b60f9f49da-tmp-dir\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.486839 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-conf-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.487012 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.487012 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-var-lib-kubelet\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.487012 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-netns\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.487012 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-bin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.487012 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhd95\" (UniqueName: \"kubernetes.io/projected/71f1519e-b03d-4ee5-9abd-b647e628531b-kube-api-access-fhd95\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.487012 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.486985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-kubelet-config\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.487012 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-dbus\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.487214 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-device-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.487214 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:49.487214 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-systemd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.487214 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-var-lib-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.487214 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-kubernetes\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.487214 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-hostroot\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.487214 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cb66043-7c25-465f-95e4-3535cb4f35ff-iptables-alerter-script\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-k8s-cni-cncf-io\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cnibin\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-run\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-tuned\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.487460 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8rb\" (UniqueName: \"kubernetes.io/projected/9193a5a4-d86a-47f5-9db2-c23851803a36-kube-api-access-8n8rb\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8drfb\" (UniqueName: \"kubernetes.io/projected/ffea2206-5878-442e-a1fd-e1a691cfbcbc-kube-api-access-8drfb\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-kubelet\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb66043-7c25-465f-95e4-3535cb4f35ff-host-slash\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-socket-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-env-overrides\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-script-lib\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-host\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-registration-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-slash\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.487720 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-netd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjf8\" (UniqueName: \"kubernetes.io/projected/b14437a8-5976-4417-99fe-481d0c2ea6b4-kube-api-access-pmjf8\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-modprobe-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-cnibin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-netns\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovn-node-metrics-cert\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-socket-dir-parent\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.487987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49dl\" (UniqueName: \"kubernetes.io/projected/414cbb04-b4d4-4840-911b-353f24025f02-kube-api-access-z49dl\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:49.488037 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-systemd-units\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-etc-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-lib-modules\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjfw\" (UniqueName: \"kubernetes.io/projected/f842fcd9-3de6-4162-a8f3-217a76df7e50-kube-api-access-mhjfw\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-daemon-config\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9193a5a4-d86a-47f5-9db2-c23851803a36-host\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-node-log\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-bin\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmxj\" (UniqueName: \"kubernetes.io/projected/42109964-a1d9-479b-b626-76b60f9f49da-kube-api-access-8gmxj\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-sys\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-system-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9193a5a4-d86a-47f5-9db2-c23851803a36-serviceca\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-sys-fs\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-conf\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-log-socket\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqj4h\" (UniqueName: \"kubernetes.io/projected/4cb66043-7c25-465f-95e4-3535cb4f35ff-kube-api-access-lqj4h\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.488564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysconfig\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.489076 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.488584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-systemd\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.514553 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514532    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:08:49.514638 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514610    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:08:49.514834 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514815    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:08:49.514900 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514891    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:08:49.514956 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514903    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:08:49.514956 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514911    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:08:49.515040 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514931    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:08:49.515040 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514955    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:08:49.515040 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514969    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d6fgf\""
Oct 30 21:08:49.515040 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.514987    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:08:49.515199 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.515043    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t284x\""
Oct 30 21:08:49.557820 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.557791    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45708598802090d7fd0a00f30bfad720.slice/crio-b8ba03cf6fc6eeae979fa03c374a2f74ea3312c94d40587e7db9b5bf1c28d8b1 WatchSource:0}: Error finding container b8ba03cf6fc6eeae979fa03c374a2f74ea3312c94d40587e7db9b5bf1c28d8b1: Status 404 returned error can't find the container with id b8ba03cf6fc6eeae979fa03c374a2f74ea3312c94d40587e7db9b5bf1c28d8b1
Oct 30 21:08:49.558878 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.558854    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd04ddc0cbac70d5dc4161808c80da69.slice/crio-ba635a1904a15e7f23e2995d8b78409c7edc657f8a5251045993696ce359aa4a WatchSource:0}: Error finding container ba635a1904a15e7f23e2995d8b78409c7edc657f8a5251045993696ce359aa4a: Status 404 returned error can't find the container with id ba635a1904a15e7f23e2995d8b78409c7edc657f8a5251045993696ce359aa4a
Oct 30 21:08:49.565726 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.565708    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:08:49.588839 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.588814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysconfig\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.588956 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.588845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-systemd\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.588956 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.588861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-tmp\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.588956 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.588888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-system-cni-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.588956 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.588933    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-system-cni-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.588956 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.588950    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysconfig\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.588958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-kubelet\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-ovn\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589017    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-systemd\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.588977    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-kubelet\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-konnectivity-ca\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589077    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-ovn\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnk9\" (UniqueName: \"kubernetes.io/projected/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kube-api-access-tbnk9\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589114    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-config\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589223    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:08:49.589255 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589264    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589303    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-cni-binary-copy\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-multus\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-multus-certs\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-etc-kubernetes\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-agent-certs\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42109964-a1d9-479b-b626-76b60f9f49da-hosts-file\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589493    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-multus-certs\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42109964-a1d9-479b-b626-76b60f9f49da-tmp-dir\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589557    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-multus\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589682    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42109964-a1d9-479b-b626-76b60f9f49da-hosts-file\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589707    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-etc-kubernetes\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-conf-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589755    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-conf-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-var-lib-kubelet\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-netns\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.589874 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-bin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhd95\" (UniqueName: \"kubernetes.io/projected/71f1519e-b03d-4ee5-9abd-b647e628531b-kube-api-access-fhd95\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-kubelet-config\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589923    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-var-lib-kubelet\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-dbus\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589953    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-bin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-device-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589980    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-netns\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.589996    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-kubelet-config\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-systemd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590039    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-device-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-var-lib-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-kubernetes\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-hostroot\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590120    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-dbus\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cb66043-7c25-465f-95e4-3535cb4f35ff-iptables-alerter-script\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.590728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-k8s-cni-cncf-io\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590221    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-var-lib-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590259    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-systemd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cnibin\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-run\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590323    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-kubernetes\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-tuned\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8rb\" (UniqueName: \"kubernetes.io/projected/9193a5a4-d86a-47f5-9db2-c23851803a36-kube-api-access-8n8rb\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590371    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cnibin\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8drfb\" (UniqueName: \"kubernetes.io/projected/ffea2206-5878-442e-a1fd-e1a691cfbcbc-kube-api-access-8drfb\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590414    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-k8s-cni-cncf-io\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590420    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-kubelet\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.591611 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.590490    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590500    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590527    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb66043-7c25-465f-95e4-3535cb4f35ff-host-slash\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.590549    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs podName:414cbb04-b4d4-4840-911b-353f24025f02 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:50.090529962 +0000 UTC m=+2.088990118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs") pod "network-metrics-daemon-p67fc" (UID: "414cbb04-b4d4-4840-911b-353f24025f02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590549    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-config\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590546    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-hostroot\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590407    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-run\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb66043-7c25-465f-95e4-3535cb4f35ff-host-slash\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-socket-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590623    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-env-overrides\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590653    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-kubelet\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-script-lib\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-host\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590789    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-host\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590833    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-konnectivity-ca\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590842    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-socket-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.592511 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-registration-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590942    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42109964-a1d9-479b-b626-76b60f9f49da-tmp-dir\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-slash\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.590996    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-slash\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591007    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-registration-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-netd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjf8\" (UniqueName: \"kubernetes.io/projected/b14437a8-5976-4417-99fe-481d0c2ea6b4-kube-api-access-pmjf8\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-modprobe-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-cnibin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-netns\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovn-node-metrics-cert\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591253    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-env-overrides\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-socket-dir-parent\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49dl\" (UniqueName: \"kubernetes.io/projected/414cbb04-b4d4-4840-911b-353f24025f02-kube-api-access-z49dl\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:49.593365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-systemd-units\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-etc-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-lib-modules\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjfw\" (UniqueName: \"kubernetes.io/projected/f842fcd9-3de6-4162-a8f3-217a76df7e50-kube-api-access-mhjfw\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-daemon-config\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9193a5a4-d86a-47f5-9db2-c23851803a36-host\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591521    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-script-lib\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591531    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-cni-binary-copy\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591575    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591581    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-netd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591594    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-cnibin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-node-log\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591624    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-netns\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591660    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-socket-dir-parent\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-bin\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmxj\" (UniqueName: \"kubernetes.io/projected/42109964-a1d9-479b-b626-76b60f9f49da-kube-api-access-8gmxj\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.594137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-sys\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-system-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9193a5a4-d86a-47f5-9db2-c23851803a36-serviceca\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591815    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-lib-modules\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-sys-fs\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591849    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591870    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-sys\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591882    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-sys-fs\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.591903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-conf\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592037    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-node-log\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592057    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592080    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-system-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.592361    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.592438    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret podName:7e7da34c-23be-4c24-977d-8ef4b007fe8f nodeName:}" failed. No retries permitted until 2026-10-30 21:08:50.092418441 +0000 UTC m=+2.090878593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret") pod "global-pull-secret-syncer-kmwrj" (UID: "7e7da34c-23be-4c24-977d-8ef4b007fe8f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592524    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-conf\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-log-socket\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqj4h\" (UniqueName: \"kubernetes.io/projected/4cb66043-7c25-465f-95e4-3535cb4f35ff-kube-api-access-lqj4h\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.594671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592618    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-etc-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592755    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-bin\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592788    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9193a5a4-d86a-47f5-9db2-c23851803a36-host\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592796    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-log-socket\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592821    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-modprobe-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.592843    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-systemd-units\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.593294    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9193a5a4-d86a-47f5-9db2-c23851803a36-serviceca\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.593670    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cb66043-7c25-465f-95e4-3535cb4f35ff-iptables-alerter-script\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.593731    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-tuned\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.593758    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-tmp\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.594063    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-agent-certs\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.594083    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.594156    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-daemon-config\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.595166 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.594488    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovn-node-metrics-cert\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.606704 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.606638    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:08:49.606704 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.606659    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:08:49.606704 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.606673    2581 projected.go:196] Error preparing data for projected volume kube-api-access-n8msz for pod openshift-network-diagnostics/network-check-target-vts2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:49.606889 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:49.606717    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz podName:6f8e3f58-0a6e-41fa-a599-9d68ae65f01d nodeName:}" failed. No retries permitted until 2026-10-30 21:08:50.106705619 +0000 UTC m=+2.105165791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n8msz" (UniqueName: "kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz") pod "network-check-target-vts2m" (UID: "6f8e3f58-0a6e-41fa-a599-9d68ae65f01d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:49.607355 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.607327    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnk9\" (UniqueName: \"kubernetes.io/projected/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kube-api-access-tbnk9\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.609125 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.608994    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8rb\" (UniqueName: \"kubernetes.io/projected/9193a5a4-d86a-47f5-9db2-c23851803a36-kube-api-access-8n8rb\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.609125 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.609077    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhd95\" (UniqueName: \"kubernetes.io/projected/71f1519e-b03d-4ee5-9abd-b647e628531b-kube-api-access-fhd95\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.609621 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.609598    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjf8\" (UniqueName: \"kubernetes.io/projected/b14437a8-5976-4417-99fe-481d0c2ea6b4-kube-api-access-pmjf8\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.609837 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.609811    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8drfb\" (UniqueName: \"kubernetes.io/projected/ffea2206-5878-442e-a1fd-e1a691cfbcbc-kube-api-access-8drfb\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.610449 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.610424    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49dl\" (UniqueName: \"kubernetes.io/projected/414cbb04-b4d4-4840-911b-353f24025f02-kube-api-access-z49dl\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:49.610544 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.610478    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmxj\" (UniqueName: \"kubernetes.io/projected/42109964-a1d9-479b-b626-76b60f9f49da-kube-api-access-8gmxj\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.610544 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.610514    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjfw\" (UniqueName: \"kubernetes.io/projected/f842fcd9-3de6-4162-a8f3-217a76df7e50-kube-api-access-mhjfw\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.610706 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.610689    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqj4h\" (UniqueName: \"kubernetes.io/projected/4cb66043-7c25-465f-95e4-3535cb4f35ff-kube-api-access-lqj4h\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.728708 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.728679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:08:49.739288 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.739264    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42109964_a1d9_479b_b626_76b60f9f49da.slice/crio-55aec019a97ffa3ba003eeb7e13ec1d88c580593e50103bfa75902174852842b WatchSource:0}: Error finding container 55aec019a97ffa3ba003eeb7e13ec1d88c580593e50103bfa75902174852842b: Status 404 returned error can't find the container with id 55aec019a97ffa3ba003eeb7e13ec1d88c580593e50103bfa75902174852842b
Oct 30 21:08:49.740246 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.740214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:49.748590 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.748568    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c22004_d04f_4ac5_ae97_224bf22bcf0e.slice/crio-88fa3b9e3541fe32c2515029381b4b437c267d40f62c8259fa033928ff6aea45 WatchSource:0}: Error finding container 88fa3b9e3541fe32c2515029381b4b437c267d40f62c8259fa033928ff6aea45: Status 404 returned error can't find the container with id 88fa3b9e3541fe32c2515029381b4b437c267d40f62c8259fa033928ff6aea45
Oct 30 21:08:49.760649 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.760631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:08:49.761744 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.761723    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:08:49.762683 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.762659    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:08:49.772854 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.772831    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d483a0_47e4_49a3_a59e_7e4fe09c8779.slice/crio-b5a59d229cc72aac1bc71a9500eb029a2de65c1d1087b44cce77d6d6c751d9df WatchSource:0}: Error finding container b5a59d229cc72aac1bc71a9500eb029a2de65c1d1087b44cce77d6d6c751d9df: Status 404 returned error can't find the container with id b5a59d229cc72aac1bc71a9500eb029a2de65c1d1087b44cce77d6d6c751d9df
Oct 30 21:08:49.773782 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.773763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w5hw9"
Oct 30 21:08:49.775470 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.775447    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf842fcd9_3de6_4162_a8f3_217a76df7e50.slice/crio-405beaeb89b647f9821e4dd1494a1edca864bb975b2059b87a4ac95d146c5a58 WatchSource:0}: Error finding container 405beaeb89b647f9821e4dd1494a1edca864bb975b2059b87a4ac95d146c5a58: Status 404 returned error can't find the container with id 405beaeb89b647f9821e4dd1494a1edca864bb975b2059b87a4ac95d146c5a58
Oct 30 21:08:49.781328 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.781308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:08:49.782979 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.782959    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f1519e_b03d_4ee5_9abd_b647e628531b.slice/crio-8f7620bb05d05c86c2409008afa20f424523fb58062ded4e213f0f4a26f453b7 WatchSource:0}: Error finding container 8f7620bb05d05c86c2409008afa20f424523fb58062ded4e213f0f4a26f453b7: Status 404 returned error can't find the container with id 8f7620bb05d05c86c2409008afa20f424523fb58062ded4e213f0f4a26f453b7
Oct 30 21:08:49.791383 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.791361    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb66043_7c25_465f_95e4_3535cb4f35ff.slice/crio-aa1c07082a8c08eea86e6a8bb896af60fde3d693bfc8d697c63ced91b62d4367 WatchSource:0}: Error finding container aa1c07082a8c08eea86e6a8bb896af60fde3d693bfc8d697c63ced91b62d4367: Status 404 returned error can't find the container with id aa1c07082a8c08eea86e6a8bb896af60fde3d693bfc8d697c63ced91b62d4367
Oct 30 21:08:49.809733 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.809713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:08:49.820019 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.819999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:08:49.820275 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.820255    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffea2206_5878_442e_a1fd_e1a691cfbcbc.slice/crio-0296c1ed3f694373535a6894beabe196ef70f25f4d72eb1b3b1cee149257f31a WatchSource:0}: Error finding container 0296c1ed3f694373535a6894beabe196ef70f25f4d72eb1b3b1cee149257f31a: Status 404 returned error can't find the container with id 0296c1ed3f694373535a6894beabe196ef70f25f4d72eb1b3b1cee149257f31a
Oct 30 21:08:49.831138 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.831117    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9193a5a4_d86a_47f5_9db2_c23851803a36.slice/crio-1f90456bf0363bc50436136b94f582c8a041d6753b55b1398824569283d64a23 WatchSource:0}: Error finding container 1f90456bf0363bc50436136b94f582c8a041d6753b55b1398824569283d64a23: Status 404 returned error can't find the container with id 1f90456bf0363bc50436136b94f582c8a041d6753b55b1398824569283d64a23
Oct 30 21:08:49.841296 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:49.841276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:08:49.850307 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:08:49.850284    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14437a8_5976_4417_99fe_481d0c2ea6b4.slice/crio-dcdc41b727c7c4d7fd94605a06c0dff5728d2ddf644c17cc2927894750769af2 WatchSource:0}: Error finding container dcdc41b727c7c4d7fd94605a06c0dff5728d2ddf644c17cc2927894750769af2: Status 404 returned error can't find the container with id dcdc41b727c7c4d7fd94605a06c0dff5728d2ddf644c17cc2927894750769af2
Oct 30 21:08:50.095728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.095635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:50.095728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.095690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:50.095951 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:50.095812    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:50.095951 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:50.095891    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs podName:414cbb04-b4d4-4840-911b-353f24025f02 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:51.095869661 +0000 UTC m=+3.094329820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs") pod "network-metrics-daemon-p67fc" (UID: "414cbb04-b4d4-4840-911b-353f24025f02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:50.095951 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:50.095812    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:50.096118 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:50.095984    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret podName:7e7da34c-23be-4c24-977d-8ef4b007fe8f nodeName:}" failed. No retries permitted until 2026-10-30 21:08:51.095965645 +0000 UTC m=+3.094425793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret") pod "global-pull-secret-syncer-kmwrj" (UID: "7e7da34c-23be-4c24-977d-8ef4b007fe8f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:50.197143 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.196428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:50.197143 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:50.196673    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:08:50.197143 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:50.196695    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:08:50.197143 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:50.196708    2581 projected.go:196] Error preparing data for projected volume kube-api-access-n8msz for pod openshift-network-diagnostics/network-check-target-vts2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:50.197143 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:50.196774    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz podName:6f8e3f58-0a6e-41fa-a599-9d68ae65f01d nodeName:}" failed. No retries permitted until 2026-10-30 21:08:51.196754092 +0000 UTC m=+3.195214255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8msz" (UniqueName: "kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz") pod "network-check-target-vts2m" (UID: "6f8e3f58-0a6e-41fa-a599-9d68ae65f01d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:50.321516 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.321330    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:08:50.437927 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.437817    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:03:50 +0000 UTC" deadline="2028-07-30 03:31:50.713342314 +0000 UTC"
Oct 30 21:08:50.437927 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.437888    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15318h23m0.27546107s"
Oct 30 21:08:50.477249 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.477168    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tsz8h" event={"ID":"42109964-a1d9-479b-b626-76b60f9f49da","Type":"ContainerStarted","Data":"55aec019a97ffa3ba003eeb7e13ec1d88c580593e50103bfa75902174852842b"}
Oct 30 21:08:50.485671 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.485633    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"dcdc41b727c7c4d7fd94605a06c0dff5728d2ddf644c17cc2927894750769af2"}
Oct 30 21:08:50.488755 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.488727    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-99v8c" event={"ID":"f842fcd9-3de6-4162-a8f3-217a76df7e50","Type":"ContainerStarted","Data":"405beaeb89b647f9821e4dd1494a1edca864bb975b2059b87a4ac95d146c5a58"}
Oct 30 21:08:50.491145 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.491116    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n9vj8" event={"ID":"a3c22004-d04f-4ac5-ae97-224bf22bcf0e","Type":"ContainerStarted","Data":"88fa3b9e3541fe32c2515029381b4b437c267d40f62c8259fa033928ff6aea45"}
Oct 30 21:08:50.495265 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.495218    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"45708598802090d7fd0a00f30bfad720","Type":"ContainerStarted","Data":"b8ba03cf6fc6eeae979fa03c374a2f74ea3312c94d40587e7db9b5bf1c28d8b1"}
Oct 30 21:08:50.498420 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.498367    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" event={"ID":"bd04ddc0cbac70d5dc4161808c80da69","Type":"ContainerStarted","Data":"ba635a1904a15e7f23e2995d8b78409c7edc657f8a5251045993696ce359aa4a"}
Oct 30 21:08:50.518915 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.518878    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q5wbv" event={"ID":"9193a5a4-d86a-47f5-9db2-c23851803a36","Type":"ContainerStarted","Data":"1f90456bf0363bc50436136b94f582c8a041d6753b55b1398824569283d64a23"}
Oct 30 21:08:50.531755 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.531668    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerStarted","Data":"0296c1ed3f694373535a6894beabe196ef70f25f4d72eb1b3b1cee149257f31a"}
Oct 30 21:08:50.557624 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.557562    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-slp8w" event={"ID":"4cb66043-7c25-465f-95e4-3535cb4f35ff","Type":"ContainerStarted","Data":"aa1c07082a8c08eea86e6a8bb896af60fde3d693bfc8d697c63ced91b62d4367"}
Oct 30 21:08:50.562106 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.562074    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w5hw9" event={"ID":"71f1519e-b03d-4ee5-9abd-b647e628531b","Type":"ContainerStarted","Data":"8f7620bb05d05c86c2409008afa20f424523fb58062ded4e213f0f4a26f453b7"}
Oct 30 21:08:50.565807 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.565745    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw" event={"ID":"d6d483a0-47e4-49a3-a59e-7e4fe09c8779","Type":"ContainerStarted","Data":"b5a59d229cc72aac1bc71a9500eb029a2de65c1d1087b44cce77d6d6c751d9df"}
Oct 30 21:08:50.735369 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:50.735241    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:08:51.103060 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:51.102949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:51.103290 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:51.103064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:51.103290 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.103136    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:51.103290 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.103180    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:51.103290 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.103232    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret podName:7e7da34c-23be-4c24-977d-8ef4b007fe8f nodeName:}" failed. No retries permitted until 2026-10-30 21:08:53.103210346 +0000 UTC m=+5.101670487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret") pod "global-pull-secret-syncer-kmwrj" (UID: "7e7da34c-23be-4c24-977d-8ef4b007fe8f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:51.103290 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.103253    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs podName:414cbb04-b4d4-4840-911b-353f24025f02 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:53.103243013 +0000 UTC m=+5.101703154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs") pod "network-metrics-daemon-p67fc" (UID: "414cbb04-b4d4-4840-911b-353f24025f02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:51.206262 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:51.204189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:51.206262 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.204375    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:08:51.206262 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.204396    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:08:51.206262 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.204417    2581 projected.go:196] Error preparing data for projected volume kube-api-access-n8msz for pod openshift-network-diagnostics/network-check-target-vts2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:51.206262 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.204480    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz podName:6f8e3f58-0a6e-41fa-a599-9d68ae65f01d nodeName:}" failed. No retries permitted until 2026-10-30 21:08:53.204458363 +0000 UTC m=+5.202918554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8msz" (UniqueName: "kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz") pod "network-check-target-vts2m" (UID: "6f8e3f58-0a6e-41fa-a599-9d68ae65f01d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:51.438323 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:51.438130    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:03:50 +0000 UTC" deadline="2028-08-10 18:32:41.59085894 +0000 UTC"
Oct 30 21:08:51.438323 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:51.438267    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15597h23m50.152595901s"
Oct 30 21:08:51.455646 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:51.455082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:51.455646 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.455226    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:08:51.455646 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:51.455556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:51.455646 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:51.455082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:51.456000 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.455679    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:08:51.456000 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:51.455774    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:08:52.362007 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:52.361905    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:08:53.121502 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:53.121132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:53.121502 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:53.121219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:53.121502 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.121359    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:53.121502 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.121429    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs podName:414cbb04-b4d4-4840-911b-353f24025f02 nodeName:}" failed. No retries permitted until 2026-10-30 21:08:57.12140614 +0000 UTC m=+9.119866292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs") pod "network-metrics-daemon-p67fc" (UID: "414cbb04-b4d4-4840-911b-353f24025f02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:53.122167 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.121817    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:53.122167 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.121860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret podName:7e7da34c-23be-4c24-977d-8ef4b007fe8f nodeName:}" failed. No retries permitted until 2026-10-30 21:08:57.121848534 +0000 UTC m=+9.120308685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret") pod "global-pull-secret-syncer-kmwrj" (UID: "7e7da34c-23be-4c24-977d-8ef4b007fe8f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:53.221521 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:53.221487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:53.221711 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.221694    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:08:53.221795 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.221713    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:08:53.221795 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.221726    2581 projected.go:196] Error preparing data for projected volume kube-api-access-n8msz for pod openshift-network-diagnostics/network-check-target-vts2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:53.221795 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.221775    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz podName:6f8e3f58-0a6e-41fa-a599-9d68ae65f01d nodeName:}" failed. No retries permitted until 2026-10-30 21:08:57.22175876 +0000 UTC m=+9.220218903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8msz" (UniqueName: "kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz") pod "network-check-target-vts2m" (UID: "6f8e3f58-0a6e-41fa-a599-9d68ae65f01d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:53.455551 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:53.455472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:53.455715 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.455615    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:08:53.456246 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:53.456046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:53.456246 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.456154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:08:53.456246 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:53.456206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:53.456552 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:53.456322    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:08:54.590372 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:54.590081    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n9vj8" event={"ID":"a3c22004-d04f-4ac5-ae97-224bf22bcf0e","Type":"ContainerStarted","Data":"faa2c14cead1a0fb6fe2ef1c1bd69a6a6bd3a4c72d8f159e0467f1a7f2da20f7"}
Oct 30 21:08:54.600229 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:54.600194    2581 generic.go:356] "Generic (PLEG): container finished" podID="45708598802090d7fd0a00f30bfad720" containerID="6dac8484e91f1e6311cc42f2a5aa993f9a66a0128171cb4621effc35665c0d6b" exitCode=0
Oct 30 21:08:54.600381 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:54.600305    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"45708598802090d7fd0a00f30bfad720","Type":"ContainerDied","Data":"6dac8484e91f1e6311cc42f2a5aa993f9a66a0128171cb4621effc35665c0d6b"}
Oct 30 21:08:54.604881 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:54.604854    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" event={"ID":"bd04ddc0cbac70d5dc4161808c80da69","Type":"ContainerStarted","Data":"e411c035255608c635dc7a98f183f95476e42244c48a44e01120f205069402c1"}
Oct 30 21:08:54.610190 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:54.610143    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="2539229ba756edc3548be46b0bfd983b14eecd62c12c2b02a663eb26f70fa170" exitCode=0
Oct 30 21:08:54.610280 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:54.610199    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerDied","Data":"2539229ba756edc3548be46b0bfd983b14eecd62c12c2b02a663eb26f70fa170"}
Oct 30 21:08:54.671171 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:54.671098    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" podStartSLOduration=4.671078768 podStartE2EDuration="4.671078768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:54.670708822 +0000 UTC m=+6.669169002" watchObservedRunningTime="2026-10-30 21:08:54.671078768 +0000 UTC m=+6.669538928"
Oct 30 21:08:55.455622 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:55.455580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:55.455823 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:55.455732    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:08:55.456154 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:55.456134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:55.456280 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:55.456258    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:08:55.456356 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:55.456333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:55.456444 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:55.456424    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:08:55.615572 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:55.615532    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"45708598802090d7fd0a00f30bfad720","Type":"ContainerStarted","Data":"69eea6ceab383a9159339028596231d8e7c6ac8402f8778ff2b80443f684407f"}
Oct 30 21:08:55.688149 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:55.688094    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" podStartSLOduration=5.688073792 podStartE2EDuration="5.688073792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:55.652821756 +0000 UTC m=+7.651281947" watchObservedRunningTime="2026-10-30 21:08:55.688073792 +0000 UTC m=+7.686533955"
Oct 30 21:08:56.621446 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:56.621359    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw" event={"ID":"d6d483a0-47e4-49a3-a59e-7e4fe09c8779","Type":"ContainerStarted","Data":"561b75cb8a069409986cc886595ab5a3a48c9623466ccf3b17d9fb32e28f553d"}
Oct 30 21:08:56.624889 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:56.624856    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q5wbv" event={"ID":"9193a5a4-d86a-47f5-9db2-c23851803a36","Type":"ContainerStarted","Data":"a783b589d8d33bb35b5a6f20cee3c3ab5b55c425fd5692e6d067eacfb95c48a6"}
Oct 30 21:08:56.658880 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:56.658830    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q5wbv" podStartSLOduration=1.820537695 podStartE2EDuration="7.658813421s" totalImagesPullingTime="5.838275726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:56.658635164 +0000 UTC m=+8.657095328" watchObservedRunningTime="2026-10-30 21:08:56.658813421 +0000 UTC m=+8.657273587"
Oct 30 21:08:56.659518 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:56.659253    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n9vj8" podStartSLOduration=3.112040021 podStartE2EDuration="7.659224248s" totalImagesPullingTime="4.547184227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:55.688552878 +0000 UTC m=+7.687013041" watchObservedRunningTime="2026-10-30 21:08:56.659224248 +0000 UTC m=+8.657684410"
Oct 30 21:08:57.155888 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.155837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:57.156082 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.155904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:57.156082 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.156061    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:57.156233 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.156129    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret podName:7e7da34c-23be-4c24-977d-8ef4b007fe8f nodeName:}" failed. No retries permitted until 2026-10-30 21:09:05.156110642 +0000 UTC m=+17.154570802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret") pod "global-pull-secret-syncer-kmwrj" (UID: "7e7da34c-23be-4c24-977d-8ef4b007fe8f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:08:57.156598 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.156576    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:57.156676 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.156641    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs podName:414cbb04-b4d4-4840-911b-353f24025f02 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:05.156622362 +0000 UTC m=+17.155082504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs") pod "network-metrics-daemon-p67fc" (UID: "414cbb04-b4d4-4840-911b-353f24025f02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:08:57.256621 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.256519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:57.256790 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.256733    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:08:57.256790 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.256756    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:08:57.256790 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.256770    2581 projected.go:196] Error preparing data for projected volume kube-api-access-n8msz for pod openshift-network-diagnostics/network-check-target-vts2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:57.256946 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.256833    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz podName:6f8e3f58-0a6e-41fa-a599-9d68ae65f01d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:05.256814751 +0000 UTC m=+17.255274903 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8msz" (UniqueName: "kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz") pod "network-check-target-vts2m" (UID: "6f8e3f58-0a6e-41fa-a599-9d68ae65f01d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:08:57.455145 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.455110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:57.455348 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.455273    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:08:57.455348 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.455308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:57.455465 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.455430    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:08:57.455514 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.455496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:57.455610 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:57.455583    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:08:57.546211 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.546002    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:57.546608 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.546591    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:57.689387 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.689200    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:57.690427 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.690203    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:08:57.894127 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:57.894096    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:08:58.453632 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:58.453518    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:08:57.894121115Z","UUID":"1918faf9-711b-4fb6-bdaf-f6ca188c8d0b","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:08:58.458083 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:58.458050    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:08:58.458263 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:58.458100    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:08:58.633080 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:58.632281    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w5hw9" event={"ID":"71f1519e-b03d-4ee5-9abd-b647e628531b","Type":"ContainerStarted","Data":"980369688a804cb75d532380ac50a8b12e0612bb6eda17a269962d73b45f5324"}
Oct 30 21:08:58.636488 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:58.636355    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw" event={"ID":"d6d483a0-47e4-49a3-a59e-7e4fe09c8779","Type":"ContainerStarted","Data":"b65c8c9c351ca9f502fbe4e29908a14eebaa943063a8cd15c12a246411a7459d"}
Oct 30 21:08:58.668786 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:58.668728    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-w5hw9" podStartSLOduration=1.240448782 podStartE2EDuration="9.668709566s" totalImagesPullingTime="8.428260784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:58.667951474 +0000 UTC m=+10.666411639" watchObservedRunningTime="2026-10-30 21:08:58.668709566 +0000 UTC m=+10.667169728"
Oct 30 21:08:59.455785 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.455483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:08:59.456264 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:59.455864    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:08:59.456343 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.456220    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:08:59.456395 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:59.456370    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:08:59.456548 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.456419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:08:59.456611 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:08:59.456544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:08:59.641841 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.641716    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="5bb50dd3dbcf1fd4f382f0c5a96780e7eaf5519e2963240de2bf3b067a66ee67" exitCode=0
Oct 30 21:08:59.641841 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.641835    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerDied","Data":"5bb50dd3dbcf1fd4f382f0c5a96780e7eaf5519e2963240de2bf3b067a66ee67"}
Oct 30 21:08:59.646092 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.645522    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw" event={"ID":"d6d483a0-47e4-49a3-a59e-7e4fe09c8779","Type":"ContainerStarted","Data":"86eba4493516429c4cfc23b5c302047c5be66757b7f16e4c5cf603656f7fad50"}
Oct 30 21:08:59.648333 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.648281    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tsz8h" event={"ID":"42109964-a1d9-479b-b626-76b60f9f49da","Type":"ContainerStarted","Data":"70112a06100b1347a485e8e3727df41e1b2873203bbc49d708c34ccd02669e69"}
Oct 30 21:08:59.753600 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.753491    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tsz8h" podStartSLOduration=1.625260621 podStartE2EDuration="10.753473468s" totalImagesPullingTime="9.128212847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:59.718226998 +0000 UTC m=+11.716687160" watchObservedRunningTime="2026-10-30 21:08:59.753473468 +0000 UTC m=+11.751933643"
Oct 30 21:08:59.754104 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:08:59.754055    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw" podStartSLOduration=1.879023651 podStartE2EDuration="10.754044443s" totalImagesPullingTime="8.875020792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:59.753316939 +0000 UTC m=+11.751777101" watchObservedRunningTime="2026-10-30 21:08:59.754044443 +0000 UTC m=+11.752504598"
Oct 30 21:09:00.653717 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:00.653622    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="affbfc60f1f1599f5f5f809092e0b27a3b0cd809018fb9d604b9aea072b8e8a9" exitCode=0
Oct 30 21:09:00.654272 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:00.654223    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerDied","Data":"affbfc60f1f1599f5f5f809092e0b27a3b0cd809018fb9d604b9aea072b8e8a9"}
Oct 30 21:09:00.656393 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:00.656363    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-slp8w" event={"ID":"4cb66043-7c25-465f-95e4-3535cb4f35ff","Type":"ContainerStarted","Data":"007f6ec555f479a31175f2b8f97064bacf75b58df010ceaaa29306fc0ba02261"}
Oct 30 21:09:00.741424 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:00.741364    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-slp8w" podStartSLOduration=2.858230742 podStartE2EDuration="11.741343712s" totalImagesPullingTime="8.88311297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:00.740437751 +0000 UTC m=+12.738897913" watchObservedRunningTime="2026-10-30 21:09:00.741343712 +0000 UTC m=+12.739803874"
Oct 30 21:09:01.454904 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:01.454832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:01.455041 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:01.454908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:01.455041 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:01.455002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:01.455041 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:01.455015    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:01.455192 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:01.455093    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:01.455192 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:01.455142    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:01.661278 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:01.661242    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="4b6649b24e275ac2f6c69cab18b8fb8606f818af13682db7e859df02fdae0d72" exitCode=0
Oct 30 21:09:01.662191 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:01.661346    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerDied","Data":"4b6649b24e275ac2f6c69cab18b8fb8606f818af13682db7e859df02fdae0d72"}
Oct 30 21:09:01.663048 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:01.663021    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-99v8c" event={"ID":"f842fcd9-3de6-4162-a8f3-217a76df7e50","Type":"ContainerStarted","Data":"adfde851fde99df2d19d6ecdee9cdfdddff3511a08a24b6b009e61f278d0cf94"}
Oct 30 21:09:01.745061 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:01.744966    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-99v8c" podStartSLOduration=1.446859492 podStartE2EDuration="12.744949315s" totalImagesPullingTime="11.298089823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:01.744350562 +0000 UTC m=+13.742810719" watchObservedRunningTime="2026-10-30 21:09:01.744949315 +0000 UTC m=+13.743409474"
Oct 30 21:09:03.454780 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:03.454746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:03.454780 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:03.454786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:03.455427 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:03.454789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:03.455427 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:03.454881    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:03.455427 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:03.454988    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:03.455427 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:03.455076    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:03.669820 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:03.669603    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerStarted","Data":"91447f050047fe0ba00608b0e5fc28ed1ec37c7594f4531345e3981a06be9c9d"}
Oct 30 21:09:04.675922 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:04.675886    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="91447f050047fe0ba00608b0e5fc28ed1ec37c7594f4531345e3981a06be9c9d" exitCode=0
Oct 30 21:09:04.676376 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:04.675944    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerDied","Data":"91447f050047fe0ba00608b0e5fc28ed1ec37c7594f4531345e3981a06be9c9d"}
Oct 30 21:09:05.226365 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.226330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:05.226551 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.226377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:05.226551 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.226478    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:05.226551 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.226548    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs podName:414cbb04-b4d4-4840-911b-353f24025f02 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:21.22652906 +0000 UTC m=+33.224989196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs") pod "network-metrics-daemon-p67fc" (UID: "414cbb04-b4d4-4840-911b-353f24025f02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:05.226701 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.226485    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:05.226701 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.226626    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret podName:7e7da34c-23be-4c24-977d-8ef4b007fe8f nodeName:}" failed. No retries permitted until 2026-10-30 21:09:21.226610519 +0000 UTC m=+33.225070670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret") pod "global-pull-secret-syncer-kmwrj" (UID: "7e7da34c-23be-4c24-977d-8ef4b007fe8f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:05.326862 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.326833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:05.326999 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.326982    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:05.327037 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.327003    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:05.327037 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.327012    2581 projected.go:196] Error preparing data for projected volume kube-api-access-n8msz for pod openshift-network-diagnostics/network-check-target-vts2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:05.327098 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.327055    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz podName:6f8e3f58-0a6e-41fa-a599-9d68ae65f01d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:21.327043623 +0000 UTC m=+33.325503759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8msz" (UniqueName: "kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz") pod "network-check-target-vts2m" (UID: "6f8e3f58-0a6e-41fa-a599-9d68ae65f01d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:05.454770 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.454741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:05.454770 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.454764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:05.454907 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.454841    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:05.454907 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.454895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:05.455037 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.455016    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:05.455128 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:05.455111    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:05.680968 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.680941    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="494638786be54395d374ed00a927a8c7cc47bac16692a2db8eae9efa8285f240" exitCode=0
Oct 30 21:09:05.681393 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.681017    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerDied","Data":"494638786be54395d374ed00a927a8c7cc47bac16692a2db8eae9efa8285f240"}
Oct 30 21:09:05.684064 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.684039    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"51bc70e78a5e382608a4b64410741b4b62d13fc8905a7c4e182ab292562518a7"}
Oct 30 21:09:05.684137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.684099    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"8c6bf56ab4de6af82697b07ae78f1a2e52612b4673f154d2929491b394d5b0e8"}
Oct 30 21:09:05.684137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.684128    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"f135784529f7c2382ab368f306760c9b4e8efe494bbb7de31daee27ab6e282a2"}
Oct 30 21:09:05.684228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.684171    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"375fa7c8f55709616b46487bfc709e5d02964c009495b290e0b048582ba27e00"}
Oct 30 21:09:05.684228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:05.684202    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"dcc330613df1520c7bf851d1c13447b2d11654c95a0b2ba41b359dd185649edd"}
Oct 30 21:09:06.688746 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:06.688711    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"b16ebd88f07b6f8680a916aa23e91b5bc073d3fc0b8defa3957b098bfedc73ff"}
Oct 30 21:09:06.691748 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:06.691718    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" event={"ID":"ffea2206-5878-442e-a1fd-e1a691cfbcbc","Type":"ContainerStarted","Data":"3ed9762257b33c207db581f8a4eb791913fe4452410c0b6629d1b098fb8a6ef1"}
Oct 30 21:09:06.763769 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:06.763730    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h2wqw" podStartSLOduration=6.691297349 podStartE2EDuration="17.763717193s" totalImagesPullingTime="11.072419844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:06.753004559 +0000 UTC m=+18.751464718" watchObservedRunningTime="2026-10-30 21:09:06.763717193 +0000 UTC m=+18.762177351"
Oct 30 21:09:07.454968 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:07.454924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:07.454968 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:07.454968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:07.455193 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:07.454970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:07.455193 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:07.455048    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:07.455193 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:07.455112    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:07.455306 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:07.455190    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:08.700961 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:08.700930    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"0b9ec7e6f67927fe4da90739ddf74046ee485cb1144b27384abc2c7245737d00"}
Oct 30 21:09:09.455069 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:09.455039    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:09.455256 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:09.455094    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:09.455256 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:09.455100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:09.455256 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:09.455183    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:09.455354 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:09.455259    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:09.455354 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:09.455317    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:10.708776 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:10.708743    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" event={"ID":"b14437a8-5976-4417-99fe-481d0c2ea6b4","Type":"ContainerStarted","Data":"679883f3e2664f377a56b0e982572d42897a604ee1459d587fa9c81539feef7c"}
Oct 30 21:09:10.709564 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:10.708998    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:09:10.723888 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:10.723865    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:09:10.782621 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:10.782586    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" podStartSLOduration=6.260133019 podStartE2EDuration="21.782575579s" totalImagesPullingTime="15.52244256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:10.782320964 +0000 UTC m=+22.780781123" watchObservedRunningTime="2026-10-30 21:09:10.782575579 +0000 UTC m=+22.781035736"
Oct 30 21:09:11.454843 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.454810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:11.454992 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.454810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:11.454992 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:11.454917    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:11.454992 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.454810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:11.455176 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:11.454999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:11.455176 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:11.455087    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:11.710892 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.710831    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:09:11.711287 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.711266    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:09:11.726704 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.726398    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:09:11.956297 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.956271    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p67fc"]
Oct 30 21:09:11.956435 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.956424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:11.956536 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:11.956519    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:11.956795 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.956775    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vts2m"]
Oct 30 21:09:11.956881 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.956870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:11.956981 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:11.956964    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:11.959131 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.959107    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kmwrj"]
Oct 30 21:09:11.959268 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:11.959216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:11.959319 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:11.959282    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:12.713118 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:12.713088    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:09:13.455188 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:13.455144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:13.455188 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:13.455195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:13.455411 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:13.455265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:13.455411 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:13.455283    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:13.455411 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:13.455336    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:13.455513 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:13.455413    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:13.560644 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:13.560600    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:09:14.728947 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:14.728862    2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6" podUID="b14437a8-5976-4417-99fe-481d0c2ea6b4" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 21:09:15.455525 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:15.455482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:15.455768 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:15.455537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:15.455768 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:15.455495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:15.455768 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:15.455608    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:15.455768 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:15.455652    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:15.455768 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:15.455726    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:17.455072 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:17.455042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:17.455480 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:17.455176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:17.455480 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:17.455192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:17.455480 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:17.455188    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:17.455480 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:17.455263    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:17.455480 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:17.455323    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:19.454492 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:19.454459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:19.454492 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:19.454484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:19.454967 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:19.454459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:19.454967 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:19.454575    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:19.454967 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:19.454638    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:19.454967 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:19.454734    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:19.874310 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:19.874239    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tsz8h_42109964-a1d9-479b-b626-76b60f9f49da/dns-node-resolver/0.log"
Oct 30 21:09:20.823737 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:20.823704    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n9vj8_a3c22004-d04f-4ac5-ae97-224bf22bcf0e/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:20.830323 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:20.830296    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q5wbv_9193a5a4-d86a-47f5-9db2-c23851803a36/node-ca/0.log"
Oct 30 21:09:21.232945 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.232875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:21.232945 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.232911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:21.233111 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.233017    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:21.233111 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.233077    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs podName:414cbb04-b4d4-4840-911b-353f24025f02 nodeName:}" failed. No retries permitted until 2026-10-30 21:09:53.233063845 +0000 UTC m=+65.231523995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs") pod "network-metrics-daemon-p67fc" (UID: "414cbb04-b4d4-4840-911b-353f24025f02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:09:21.233111 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.233080    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:21.233233 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.233126    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret podName:7e7da34c-23be-4c24-977d-8ef4b007fe8f nodeName:}" failed. No retries permitted until 2026-10-30 21:09:53.233112823 +0000 UTC m=+65.231572963 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret") pod "global-pull-secret-syncer-kmwrj" (UID: "7e7da34c-23be-4c24-977d-8ef4b007fe8f") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:09:21.333909 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.333878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:21.334069 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.334024    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:09:21.334069 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.334043    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:09:21.334069 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.334052    2581 projected.go:196] Error preparing data for projected volume kube-api-access-n8msz for pod openshift-network-diagnostics/network-check-target-vts2m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:21.334194 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.334096    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz podName:6f8e3f58-0a6e-41fa-a599-9d68ae65f01d nodeName:}" failed. No retries permitted until 2026-10-30 21:09:53.334083598 +0000 UTC m=+65.332543734 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8msz" (UniqueName: "kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz") pod "network-check-target-vts2m" (UID: "6f8e3f58-0a6e-41fa-a599-9d68ae65f01d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:09:21.393182 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.393139    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n9vj8_a3c22004-d04f-4ac5-ae97-224bf22bcf0e/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:21.454633 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.454600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:21.454800 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.454668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:21.454800 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.454771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:21.454800 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.454791    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:21.454959 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.454852    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:21.454959 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:21.454892    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:21.563551 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.563479    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n9vj8_a3c22004-d04f-4ac5-ae97-224bf22bcf0e/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:21.718618 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.718585    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n9vj8_a3c22004-d04f-4ac5-ae97-224bf22bcf0e/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:21.873741 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:21.873662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n9vj8_a3c22004-d04f-4ac5-ae97-224bf22bcf0e/konnectivity-agent/0.log" limit=1024
Oct 30 21:09:23.455381 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:23.455354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:23.455381 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:23.455383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:23.455799 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:23.455473    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:23.455799 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:23.455521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:23.455799 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:23.455602    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:23.455799 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:23.455676    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:25.455151 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:25.455112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:25.455151 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:25.455132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:25.455621 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:25.455249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:25.455621 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:25.455267    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:25.455621 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:25.455316    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:25.455621 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:25.455421    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:27.455443 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:27.455413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:27.455835 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:27.455414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:27.455835 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:27.455520    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:27.455835 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:27.455414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:27.455835 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:27.455579    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:27.455835 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:27.455652    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:29.454999 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:29.454970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:29.454999 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:29.455003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:29.455504 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:29.455011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:29.455504 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:29.455100    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:29.455504 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:29.455214    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:29.455504 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:29.455309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:31.454559 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:31.454521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:31.455190 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:31.454569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:31.455190 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:31.454633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:31.455190 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:31.454657    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:31.455190 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:31.454707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:31.455190 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:31.454773    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:33.455168 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:33.455136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:33.455657 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:33.455178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:33.455657 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:33.455192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:33.455657 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:33.455297    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:33.455657 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:33.455395    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:33.455657 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:33.455452    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:35.455195 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.455145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:35.455596 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.455202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:35.455596 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.455252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:35.455596 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:35.455269    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmwrj" podUID="7e7da34c-23be-4c24-977d-8ef4b007fe8f"
Oct 30 21:09:35.455596 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:35.455319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vts2m" podUID="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d"
Oct 30 21:09:35.455596 ip-10-0-129-84 kubenswrapper[2581]: E1030 21:09:35.455417    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p67fc" podUID="414cbb04-b4d4-4840-911b-353f24025f02"
Oct 30 21:09:35.818024 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.817993    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeReady"
Oct 30 21:09:35.818185 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.818135    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:09:35.882500 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.882460    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gvjst"]
Oct 30 21:09:35.885699 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.885680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:35.890480 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.890457    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zsrxr"]
Oct 30 21:09:35.893286 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.893266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:35.899268 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.899249    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:09:35.899358 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.899342    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8rdf8\""
Oct 30 21:09:35.900714 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.900692    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:09:35.900714 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.900698    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:09:35.900823 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.900733    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:09:35.912128 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.912108    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-krgh5"]
Oct 30 21:09:35.914838 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.914818    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gvjst"]
Oct 30 21:09:35.914917 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.914857    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zsrxr"]
Oct 30 21:09:35.914975 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.914926    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:09:35.939062 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.939037    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-krgh5"]
Oct 30 21:09:35.968083 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.968027    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:09:35.968239 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.968224    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:09:35.968286 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.968270    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:09:35.999737 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.999712    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w6j9l\""
Oct 30 21:09:35.999838 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.999744    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:09:35.999838 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.999764    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6qvgx\""
Oct 30 21:09:35.999838 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:35.999772    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:09:36.042377 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dbdcb7d-b158-484a-832b-30fa8cc62c53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.042474 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6427a41a-4ada-4a89-a06a-b28b84815333-metrics-tls\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.042474 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d2612a3-2331-4145-81bc-1c2eea9a37a5-cert\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:09:36.042557 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dbdcb7d-b158-484a-832b-30fa8cc62c53-crio-socket\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.042557 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dbdcb7d-b158-484a-832b-30fa8cc62c53-data-volume\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.042557 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6drm\" (UniqueName: \"kubernetes.io/projected/9d2612a3-2331-4145-81bc-1c2eea9a37a5-kube-api-access-p6drm\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:09:36.042657 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686tm\" (UniqueName: \"kubernetes.io/projected/6427a41a-4ada-4a89-a06a-b28b84815333-kube-api-access-686tm\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.042712 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28qc\" (UniqueName: \"kubernetes.io/projected/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-api-access-m28qc\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.042747 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6427a41a-4ada-4a89-a06a-b28b84815333-tmp-dir\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.042793 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.042827 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.042800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6427a41a-4ada-4a89-a06a-b28b84815333-config-volume\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.098446 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.098421    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-786f578d97-qkqng"]
Oct 30 21:09:36.101343 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.101325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.114510 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.114483    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:09:36.114696 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.114669    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:09:36.114696 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.114671    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:09:36.114835 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.114745    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:09:36.114835 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.114751    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v2rbx\""
Oct 30 21:09:36.114835 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.114752    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:09:36.114835 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.114753    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:09:36.114957 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.114870    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:09:36.122433 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.122415    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786f578d97-qkqng"]
Oct 30 21:09:36.143223 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6427a41a-4ada-4a89-a06a-b28b84815333-config-volume\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.143324 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dbdcb7d-b158-484a-832b-30fa8cc62c53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.143324 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6427a41a-4ada-4a89-a06a-b28b84815333-metrics-tls\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.143324 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d2612a3-2331-4145-81bc-1c2eea9a37a5-cert\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:09:36.143324 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dbdcb7d-b158-484a-832b-30fa8cc62c53-crio-socket\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.143324 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dbdcb7d-b158-484a-832b-30fa8cc62c53-data-volume\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.143567 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6drm\" (UniqueName: \"kubernetes.io/projected/9d2612a3-2331-4145-81bc-1c2eea9a37a5-kube-api-access-p6drm\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:09:36.143567 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686tm\" (UniqueName: \"kubernetes.io/projected/6427a41a-4ada-4a89-a06a-b28b84815333-kube-api-access-686tm\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.143567 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28qc\" (UniqueName: \"kubernetes.io/projected/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-api-access-m28qc\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.143567 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6427a41a-4ada-4a89-a06a-b28b84815333-tmp-dir\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.143567 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143473    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dbdcb7d-b158-484a-832b-30fa8cc62c53-crio-socket\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.143567 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.143826 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143697    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dbdcb7d-b158-484a-832b-30fa8cc62c53-data-volume\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.143826 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.143811    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6427a41a-4ada-4a89-a06a-b28b84815333-tmp-dir\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.144165 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.144137    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.144251 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.144148    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6427a41a-4ada-4a89-a06a-b28b84815333-config-volume\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.146841 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.146816    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dbdcb7d-b158-484a-832b-30fa8cc62c53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.146919 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.146884    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6427a41a-4ada-4a89-a06a-b28b84815333-metrics-tls\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.147059 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.147042    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d2612a3-2331-4145-81bc-1c2eea9a37a5-cert\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:09:36.163045 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.163021    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28qc\" (UniqueName: \"kubernetes.io/projected/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-api-access-m28qc\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.164127 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.164109    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6drm\" (UniqueName: \"kubernetes.io/projected/9d2612a3-2331-4145-81bc-1c2eea9a37a5-kube-api-access-p6drm\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:09:36.164267 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.164246    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686tm\" (UniqueName: \"kubernetes.io/projected/6427a41a-4ada-4a89-a06a-b28b84815333-kube-api-access-686tm\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.202992 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.202968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:09:36.208062 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.208045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:36.226953 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.226872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:09:36.247489 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.247451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.247645 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.247500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.247645 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.247543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.247645 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.247613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.247805 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.247641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9xhq\" (UniqueName: \"kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.247805 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.247670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.247805 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.247714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.348936 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.348902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349062 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.348964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349062 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.348980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9xhq\" (UniqueName: \"kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349062 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.348998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349062 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.349024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349062 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.349048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349332 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.349077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349517 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.349494    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349725 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.349707    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349773 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.349728    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.349773 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.349709    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.351222 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.351204    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.351377 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.351362    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.366333 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.366310    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9xhq\" (UniqueName: \"kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.411258 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.411234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:36.512672 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.512614    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zsrxr"]
Oct 30 21:09:36.518883 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:36.518860    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6427a41a_4ada_4a89_a06a_b28b84815333.slice/crio-28f1fd56fe18a3f82bcfb1ddf124759362108a7b5bc88d6848b1ddb927d9fd73 WatchSource:0}: Error finding container 28f1fd56fe18a3f82bcfb1ddf124759362108a7b5bc88d6848b1ddb927d9fd73: Status 404 returned error can't find the container with id 28f1fd56fe18a3f82bcfb1ddf124759362108a7b5bc88d6848b1ddb927d9fd73
Oct 30 21:09:36.526705 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.526680    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gvjst"]
Oct 30 21:09:36.531720 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:36.531690    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbdcb7d_b158_484a_832b_30fa8cc62c53.slice/crio-bb9c7d5488769aa958b70d25c261ce711c886f6895fb7c9ca0cba97c70f3a17b WatchSource:0}: Error finding container bb9c7d5488769aa958b70d25c261ce711c886f6895fb7c9ca0cba97c70f3a17b: Status 404 returned error can't find the container with id bb9c7d5488769aa958b70d25c261ce711c886f6895fb7c9ca0cba97c70f3a17b
Oct 30 21:09:36.546848 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.546826    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-krgh5"]
Oct 30 21:09:36.556061 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:36.556030    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2612a3_2331_4145_81bc_1c2eea9a37a5.slice/crio-f03898478d98a930901a9bf7be7f94d6b43c50d3cdb3dc3a31a90613fef0e1d6 WatchSource:0}: Error finding container f03898478d98a930901a9bf7be7f94d6b43c50d3cdb3dc3a31a90613fef0e1d6: Status 404 returned error can't find the container with id f03898478d98a930901a9bf7be7f94d6b43c50d3cdb3dc3a31a90613fef0e1d6
Oct 30 21:09:36.703686 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.703653    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786f578d97-qkqng"]
Oct 30 21:09:36.708687 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:36.708655    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35b113e_843a_429a_883c_053ea13d9d0a.slice/crio-52be8e7c3a65736e0349b24059a842cee919e3da1a15b4535772212373c2bc5c WatchSource:0}: Error finding container 52be8e7c3a65736e0349b24059a842cee919e3da1a15b4535772212373c2bc5c: Status 404 returned error can't find the container with id 52be8e7c3a65736e0349b24059a842cee919e3da1a15b4535772212373c2bc5c
Oct 30 21:09:36.761020 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.760980    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvjst" event={"ID":"4dbdcb7d-b158-484a-832b-30fa8cc62c53","Type":"ContainerStarted","Data":"262708e0fa99d6832b67f7ae4d598b84ad5ae716abd9b2d770ca364f0c9c6040"}
Oct 30 21:09:36.761180 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.761042    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvjst" event={"ID":"4dbdcb7d-b158-484a-832b-30fa8cc62c53","Type":"ContainerStarted","Data":"bb9c7d5488769aa958b70d25c261ce711c886f6895fb7c9ca0cba97c70f3a17b"}
Oct 30 21:09:36.762095 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.762068    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zsrxr" event={"ID":"6427a41a-4ada-4a89-a06a-b28b84815333","Type":"ContainerStarted","Data":"28f1fd56fe18a3f82bcfb1ddf124759362108a7b5bc88d6848b1ddb927d9fd73"}
Oct 30 21:09:36.763318 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.763270    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786f578d97-qkqng" event={"ID":"c35b113e-843a-429a-883c-053ea13d9d0a","Type":"ContainerStarted","Data":"52be8e7c3a65736e0349b24059a842cee919e3da1a15b4535772212373c2bc5c"}
Oct 30 21:09:36.764329 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:36.764311    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-krgh5" event={"ID":"9d2612a3-2331-4145-81bc-1c2eea9a37a5","Type":"ContainerStarted","Data":"f03898478d98a930901a9bf7be7f94d6b43c50d3cdb3dc3a31a90613fef0e1d6"}
Oct 30 21:09:37.455287 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.455200    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:37.455666 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.455646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:37.456121 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.456096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:37.469119 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.469095    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:09:37.469261 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.469203    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2bqgz\""
Oct 30 21:09:37.469261 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.469252    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:09:37.469371 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.469298    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:09:37.469472 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.469456    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbg6s\""
Oct 30 21:09:37.469546 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.469520    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:09:37.469627 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.469612    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hscbl\""
Oct 30 21:09:37.772464 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.772376    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvjst" event={"ID":"4dbdcb7d-b158-484a-832b-30fa8cc62c53","Type":"ContainerStarted","Data":"2a8951bc2f0fa3f4b7b70694bb9fe45d112d3ff89bb8748d18d2618cbfc57081"}
Oct 30 21:09:37.899603 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:37.899523    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tsz8h_42109964-a1d9-479b-b626-76b60f9f49da/dns-node-resolver/0.log"
Oct 30 21:09:38.858181 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.857953    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"]
Oct 30 21:09:38.862546 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.862525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:38.875662 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.875630    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:09:38.875662 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.875668    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-48zwb\""
Oct 30 21:09:38.875852 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.875742    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:09:38.875955 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.875939    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:09:38.876053 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.876040    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:09:38.876151 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.876103    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:09:38.886737 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.886713    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"]
Oct 30 21:09:38.970918 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.970700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgkv7\" (UniqueName: \"kubernetes.io/projected/66b15cb4-0694-40c5-aeac-939d1635a93f-kube-api-access-rgkv7\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:38.970918 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.970736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:38.970918 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.970762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:38.970918 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:38.970786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66b15cb4-0694-40c5-aeac-939d1635a93f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.071678 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.071636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgkv7\" (UniqueName: \"kubernetes.io/projected/66b15cb4-0694-40c5-aeac-939d1635a93f-kube-api-access-rgkv7\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.071865 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.071698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.071865 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.071735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.071865 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.071776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66b15cb4-0694-40c5-aeac-939d1635a93f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.072639 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.072605    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66b15cb4-0694-40c5-aeac-939d1635a93f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.075077 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.075019    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.081645 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.081618    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.093680 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.093653    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgkv7\" (UniqueName: \"kubernetes.io/projected/66b15cb4-0694-40c5-aeac-939d1635a93f-kube-api-access-rgkv7\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.095246 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.095219    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q5wbv_9193a5a4-d86a-47f5-9db2-c23851803a36/node-ca/0.log"
Oct 30 21:09:39.177663 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.177641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:09:39.480911 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.480877    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"]
Oct 30 21:09:39.486133 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:39.486103    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b15cb4_0694_40c5_aeac_939d1635a93f.slice/crio-4d916d909e3acfe19ef7a30c62362a9c2b709d895a44dafb55c4d9f20a76dd2c WatchSource:0}: Error finding container 4d916d909e3acfe19ef7a30c62362a9c2b709d895a44dafb55c4d9f20a76dd2c: Status 404 returned error can't find the container with id 4d916d909e3acfe19ef7a30c62362a9c2b709d895a44dafb55c4d9f20a76dd2c
Oct 30 21:09:39.780794 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.780117    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvjst" event={"ID":"4dbdcb7d-b158-484a-832b-30fa8cc62c53","Type":"ContainerStarted","Data":"9868693010b5916a81f0de4af07bcbf480ac6df8b3e4c231563e2a9137edb42f"}
Oct 30 21:09:39.783691 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.783636    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zsrxr" event={"ID":"6427a41a-4ada-4a89-a06a-b28b84815333","Type":"ContainerStarted","Data":"6d1f38104a0dfdd85ced1920468973d476ac1d2afb8d4161a05d6f3f989fa691"}
Oct 30 21:09:39.783691 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.783694    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:39.783886 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.783719    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zsrxr" event={"ID":"6427a41a-4ada-4a89-a06a-b28b84815333","Type":"ContainerStarted","Data":"9499be071a78d01522fbfdcfab50eb943d57e86a9ee82be56fd03e39b9ee4c85"}
Oct 30 21:09:39.785067 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.785038    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst" event={"ID":"66b15cb4-0694-40c5-aeac-939d1635a93f","Type":"ContainerStarted","Data":"4d916d909e3acfe19ef7a30c62362a9c2b709d895a44dafb55c4d9f20a76dd2c"}
Oct 30 21:09:39.816821 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.816768    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gvjst" podStartSLOduration=2.329311412 podStartE2EDuration="4.81674985s" totalImagesPullingTime="2.487438438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:39.8163684 +0000 UTC m=+51.814828568" watchObservedRunningTime="2026-10-30 21:09:39.81674985 +0000 UTC m=+51.815210014"
Oct 30 21:09:39.854815 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:39.854767    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zsrxr" podStartSLOduration=2.610290591 podStartE2EDuration="4.85474941s" totalImagesPullingTime="2.244458819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:39.854381741 +0000 UTC m=+51.852841920" watchObservedRunningTime="2026-10-30 21:09:39.85474941 +0000 UTC m=+51.853209570"
Oct 30 21:09:40.790911 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:40.790875    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst" event={"ID":"66b15cb4-0694-40c5-aeac-939d1635a93f","Type":"ContainerStarted","Data":"fd1e9c660cb1c69bfcc82df407a54bb15d72dc080fdc618fb04e8d524ea07772"}
Oct 30 21:09:40.791342 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:40.790978    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst" event={"ID":"66b15cb4-0694-40c5-aeac-939d1635a93f","Type":"ContainerStarted","Data":"89cea9034c10bf9e1422e087a822868b09c8a61fc6f286a5ec8916e3ccfc0c42"}
Oct 30 21:09:40.792535 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:40.792499    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786f578d97-qkqng" event={"ID":"c35b113e-843a-429a-883c-053ea13d9d0a","Type":"ContainerStarted","Data":"ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278"}
Oct 30 21:09:40.793905 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:40.793878    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-krgh5" event={"ID":"9d2612a3-2331-4145-81bc-1c2eea9a37a5","Type":"ContainerStarted","Data":"79ae7fdd370e3c9ff799e653cbaece1b857cd692e8411b77c578b8be99798f9f"}
Oct 30 21:09:40.826877 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:40.826837    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst" podStartSLOduration=1.673460701 podStartE2EDuration="2.826827075s" totalImagesPullingTime="1.153366374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:40.826055654 +0000 UTC m=+52.824515811" watchObservedRunningTime="2026-10-30 21:09:40.826827075 +0000 UTC m=+52.825287225"
Oct 30 21:09:40.860080 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:40.860037    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-krgh5" podStartSLOduration=2.624342166 podStartE2EDuration="5.860027773s" totalImagesPullingTime="3.235685607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:40.858928329 +0000 UTC m=+52.857388486" watchObservedRunningTime="2026-10-30 21:09:40.860027773 +0000 UTC m=+52.858487931"
Oct 30 21:09:40.895841 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:40.895758    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-786f578d97-qkqng" podStartSLOduration=1.393807862 podStartE2EDuration="4.895744696s" totalImagesPullingTime="3.501936834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:40.894731282 +0000 UTC m=+52.893191440" watchObservedRunningTime="2026-10-30 21:09:40.895744696 +0000 UTC m=+52.894204854"
Oct 30 21:09:42.507810 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.507778    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7bnkf"]
Oct 30 21:09:42.511052 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.511027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.532908 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.532880    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28pwt\""
Oct 30 21:09:42.532908 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.532891    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:09:42.533673 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.533658    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:09:42.578808 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.578778    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:09:42.708117 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-textfile\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.708117 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.708370 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.708370 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-sys\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.708456 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-wtmp\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.708456 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-tls\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.708520 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-metrics-client-ca\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.708520 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-root\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.708607 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.708586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5b4z\" (UniqueName: \"kubernetes.io/projected/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-kube-api-access-q5b4z\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809273 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809273 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-sys\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-wtmp\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-tls\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809365    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-sys\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-metrics-client-ca\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809446    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-wtmp\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-root\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809501    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-root\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5b4z\" (UniqueName: \"kubernetes.io/projected/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-kube-api-access-q5b4z\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809555 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-textfile\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809978 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.809978 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.809940    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-textfile\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.810137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.810019    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.810137 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.810032    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-metrics-client-ca\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.811411 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.811389    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.811559 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.811544    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-tls\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:42.826945 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:42.826922    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5b4z\" (UniqueName: \"kubernetes.io/projected/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-kube-api-access-q5b4z\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:43.125989 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:43.125909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:09:43.136891 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:43.136863    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e1ba50_dfdf_4627_8ab6_97d9b1598d51.slice/crio-72802bc74c71245f9e1cbbc7a0e887e28a22016d3d23f0eb9baa94c28341d963 WatchSource:0}: Error finding container 72802bc74c71245f9e1cbbc7a0e887e28a22016d3d23f0eb9baa94c28341d963: Status 404 returned error can't find the container with id 72802bc74c71245f9e1cbbc7a0e887e28a22016d3d23f0eb9baa94c28341d963
Oct 30 21:09:43.808347 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:43.808307    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bnkf" event={"ID":"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51","Type":"ContainerStarted","Data":"72802bc74c71245f9e1cbbc7a0e887e28a22016d3d23f0eb9baa94c28341d963"}
Oct 30 21:09:44.728199 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:44.728168    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:09:44.812604 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:44.812565    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4e1ba50-dfdf-4627-8ab6-97d9b1598d51" containerID="e2aa269852dbc58f8ff39b716369239c3990517bedf8921ecc76c0ea47f9e26d" exitCode=0
Oct 30 21:09:44.812988 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:44.812657    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bnkf" event={"ID":"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51","Type":"ContainerDied","Data":"e2aa269852dbc58f8ff39b716369239c3990517bedf8921ecc76c0ea47f9e26d"}
Oct 30 21:09:45.817859 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:45.817822    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bnkf" event={"ID":"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51","Type":"ContainerStarted","Data":"c16140aadbd02b70f0c1476b45187298e889ad7e2bee3defa0975a5de5001891"}
Oct 30 21:09:45.818256 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:45.817879    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bnkf" event={"ID":"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51","Type":"ContainerStarted","Data":"42191426ec0387cb9d45dd51bf94b5dc49407193b236a7d34e930520e649fa0b"}
Oct 30 21:09:45.876531 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:45.876460    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7bnkf" podStartSLOduration=2.801450507 podStartE2EDuration="3.876448067s" totalImagesPullingTime="1.008885808s" totalInitContainerRuntime="66.111752ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:45.865794497 +0000 UTC m=+57.864254649" watchObservedRunningTime="2026-10-30 21:09:45.876448067 +0000 UTC m=+57.874908225"
Oct 30 21:09:46.412122 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:46.412083    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:46.412319 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:46.412169    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:46.416899 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:46.416876    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:46.826747 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:46.826677    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:09:47.809082 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.809046    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-59b7858fc8-fr72g"]
Oct 30 21:09:47.812252 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.812236    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.827322 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.827297    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59b7858fc8-fr72g"]
Oct 30 21:09:47.836979 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.836958    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 21:09:47.837096 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.837015    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 21:09:47.837096 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.837018    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 21:09:47.837096 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.837018    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 21:09:47.837212 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.837031    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 21:09:47.837212 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.837142    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sw2nt\""
Oct 30 21:09:47.844801 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.844779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.844854 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.844810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44gpq\" (UniqueName: \"kubernetes.io/projected/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-kube-api-access-44gpq\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.844854 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.844828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.844958 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.844939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.844995 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.844971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-metrics-client-ca\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.845293 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.845271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.845371 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.845331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-serving-certs-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.845414 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.845389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-federate-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.850447 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.850429    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 21:09:47.945963 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.945933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.945963 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.945964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-metrics-client-ca\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.946149 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.946002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.946149 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.946027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-serving-certs-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.946149 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.946066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-federate-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.946149 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.946085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.946149 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.946104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44gpq\" (UniqueName: \"kubernetes.io/projected/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-kube-api-access-44gpq\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.946149 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.946135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.946905 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.946852    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-metrics-client-ca\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.947025 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.946913    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-serving-certs-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.947571 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.947541    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.948594 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.948574    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.948822 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.948800    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.948869 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.948841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-federate-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.948869 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.948861    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:47.963323 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:47.963300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44gpq\" (UniqueName: \"kubernetes.io/projected/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-kube-api-access-44gpq\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:48.127586 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:48.127507    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:09:48.453570 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:48.453536    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59b7858fc8-fr72g"]
Oct 30 21:09:48.472216 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:48.472189    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf69dd8_b830_4e07_85d0_4950dcd2c9cf.slice/crio-cfab3c5fa3eaa15d46d4a0b62e7bcde86183e3efd58615c9fcc6e2ecb96face9 WatchSource:0}: Error finding container cfab3c5fa3eaa15d46d4a0b62e7bcde86183e3efd58615c9fcc6e2ecb96face9: Status 404 returned error can't find the container with id cfab3c5fa3eaa15d46d4a0b62e7bcde86183e3efd58615c9fcc6e2ecb96face9
Oct 30 21:09:48.828774 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:48.828692    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g" event={"ID":"adf69dd8-b830-4e07-85d0-4950dcd2c9cf","Type":"ContainerStarted","Data":"cfab3c5fa3eaa15d46d4a0b62e7bcde86183e3efd58615c9fcc6e2ecb96face9"}
Oct 30 21:09:49.405832 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.405787    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:09:49.410500 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.410473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.435373 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435346    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3q4vg4fcnnq04\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435443    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435454    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435443    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435491    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435497    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435526    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435530    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4xf7\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435550    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435596    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435605    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435614    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:09:49.435655 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435648    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:09:49.436182 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.435704    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:09:49.456777 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.456745    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:09:49.456777 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.456779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457031 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.456826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457031 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.456898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457031 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.456984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457031 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457228 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457439 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457439 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457439 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lfx\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457439 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457439 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457439 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457439 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.457439 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.457424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.458896 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.458823    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:09:49.458896 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.458862    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:09:49.460123 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.460098    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:09:49.558728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.558692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.558728 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.558726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.558948 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.558745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lfx\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.558948 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.558860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559041 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559090 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559142 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559223 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559285 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559335 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559335 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559436 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559436 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559436 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559436 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559615 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559615 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559615 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559615 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559615 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.559809 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.559788    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.563497 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.562253    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.563497 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.562735    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.564818 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.563915    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.564818 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.563982    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.564818 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.564460    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.565070 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.564934    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.565785 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.565442    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.565785 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.565550    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.565961 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.565808    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.569061 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.569024    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.569230 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.569205    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.569822 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.569793    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.569945 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.569921    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.570240 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.570207    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.570921 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.570871    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.570921 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.570882    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.571242 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.571221    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.571793 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.571775    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.577672 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.577648    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lfx\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.742724 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.742642    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:49.802593 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:49.800211    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:09:50.081442 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:50.081347    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:09:50.094184 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:50.094133    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bea9add_f465_49bf_ac0f_c2586e641917.slice/crio-e9f44828ea5416868e085ae70cc40b9a944612b2654d11fd3b5a2d4337c75b80 WatchSource:0}: Error finding container e9f44828ea5416868e085ae70cc40b9a944612b2654d11fd3b5a2d4337c75b80: Status 404 returned error can't find the container with id e9f44828ea5416868e085ae70cc40b9a944612b2654d11fd3b5a2d4337c75b80
Oct 30 21:09:50.837113 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:50.837069    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerStarted","Data":"e9f44828ea5416868e085ae70cc40b9a944612b2654d11fd3b5a2d4337c75b80"}
Oct 30 21:09:50.839011 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:50.838980    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g" event={"ID":"adf69dd8-b830-4e07-85d0-4950dcd2c9cf","Type":"ContainerStarted","Data":"5d3d531632f36c94cb7a5615a479f48c2c3eac2dc180ee11359d9dbcf8e59aca"}
Oct 30 21:09:51.844641 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:51.844598    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g" event={"ID":"adf69dd8-b830-4e07-85d0-4950dcd2c9cf","Type":"ContainerStarted","Data":"6d65647af530607a5aa834ed5a82b23d7e2f643a8a17c21880082027855a7777"}
Oct 30 21:09:51.845087 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:51.844671    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g" event={"ID":"adf69dd8-b830-4e07-85d0-4950dcd2c9cf","Type":"ContainerStarted","Data":"de7eb0167a1751e37b72a103019e7dd14efa6c80be58d5368fcae92e7a7656ad"}
Oct 30 21:09:51.846244 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:51.846217    2581 generic.go:356] "Generic (PLEG): container finished" podID="8bea9add-f465-49bf-ac0f-c2586e641917" containerID="e0702c68f1e9803de7ce4b24d262a10d472f1714a011375fadcce82c7bafa920" exitCode=0
Oct 30 21:09:51.846388 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:51.846278    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerDied","Data":"e0702c68f1e9803de7ce4b24d262a10d472f1714a011375fadcce82c7bafa920"}
Oct 30 21:09:51.883319 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:51.883255    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g" podStartSLOduration=2.211188528 podStartE2EDuration="4.883242499s" totalImagesPullingTime="2.672053971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:51.881839455 +0000 UTC m=+63.880299614" watchObservedRunningTime="2026-10-30 21:09:51.883242499 +0000 UTC m=+63.881702656"
Oct 30 21:09:53.294448 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.293989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:53.294448 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.294075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:53.306292 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.306262    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:09:53.306292 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.306262    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:09:53.326601 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.326564    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:53.327319 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.327259    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:53.383122 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.383087    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2bqgz\""
Oct 30 21:09:53.392187 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.392142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:09:53.394645 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.394617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:53.403910 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.403884    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hscbl\""
Oct 30 21:09:53.407329 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.407306    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:09:53.412302 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.412279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:09:53.427838 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.427454    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:09:53.430041 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.430011    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:53.692433 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.692396    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbg6s\""
Oct 30 21:09:53.695958 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.695921    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kmwrj"]
Oct 30 21:09:53.700256 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.700024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:53.701892 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:53.701865    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7da34c_23be_4c24_977d_8ef4b007fe8f.slice/crio-382d88ffbbd482aa8f780935f1b7c598c22babf4fd49491a4b55293069eb58ea WatchSource:0}: Error finding container 382d88ffbbd482aa8f780935f1b7c598c22babf4fd49491a4b55293069eb58ea: Status 404 returned error can't find the container with id 382d88ffbbd482aa8f780935f1b7c598c22babf4fd49491a4b55293069eb58ea
Oct 30 21:09:53.727818 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.727793    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p67fc"]
Oct 30 21:09:53.855068 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.854993    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p67fc" event={"ID":"414cbb04-b4d4-4840-911b-353f24025f02","Type":"ContainerStarted","Data":"16ef3d349fba818afd960df5f45e40bae40443b675c13a78a763c84bd568cd0a"}
Oct 30 21:09:53.856356 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:53.856316    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kmwrj" event={"ID":"7e7da34c-23be-4c24-977d-8ef4b007fe8f","Type":"ContainerStarted","Data":"382d88ffbbd482aa8f780935f1b7c598c22babf4fd49491a4b55293069eb58ea"}
Oct 30 21:09:54.003302 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:54.003220    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vts2m"]
Oct 30 21:09:54.010522 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:54.010487    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8e3f58_0a6e_41fa_a599_9d68ae65f01d.slice/crio-48e104e518efb95b36d01295ccfad477854de2eed39c12f3ef19b9a056ca5034 WatchSource:0}: Error finding container 48e104e518efb95b36d01295ccfad477854de2eed39c12f3ef19b9a056ca5034: Status 404 returned error can't find the container with id 48e104e518efb95b36d01295ccfad477854de2eed39c12f3ef19b9a056ca5034
Oct 30 21:09:54.865190 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:54.865104    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vts2m" event={"ID":"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d","Type":"ContainerStarted","Data":"48e104e518efb95b36d01295ccfad477854de2eed39c12f3ef19b9a056ca5034"}
Oct 30 21:09:55.882106 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:55.882025    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerStarted","Data":"4d85d9becdf286edd7c10424da4af35274956952ec5b6ac258238c60db5a23b1"}
Oct 30 21:09:55.882568 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:55.882108    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerStarted","Data":"45bd60f4490cad10a4f87976d7566c85eda6eec05d66724c28708fa2427019f9"}
Oct 30 21:09:55.885426 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:55.884671    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p67fc" event={"ID":"414cbb04-b4d4-4840-911b-353f24025f02","Type":"ContainerStarted","Data":"fca3ee54b73d539f1515aa9ae632a8c4c36ebac87e6d737ebe4043bf3a171b37"}
Oct 30 21:09:55.885426 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:55.884712    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p67fc" event={"ID":"414cbb04-b4d4-4840-911b-353f24025f02","Type":"ContainerStarted","Data":"d07c8e6aca3329291b9c2328336f8f0036b11fb2253ab0b097f88f0472d9a790"}
Oct 30 21:09:55.928937 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:55.928877    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p67fc" podStartSLOduration=65.714513043 podStartE2EDuration="1m6.928857637s" totalImagesPullingTime="1.214344594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:55.927044005 +0000 UTC m=+67.925504164" watchObservedRunningTime="2026-10-30 21:09:55.928857637 +0000 UTC m=+67.927317797"
Oct 30 21:09:57.986747 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:57.985288    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5fd76869-cqkmj"]
Oct 30 21:09:57.989993 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:57.989966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.028761 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.028715    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5fd76869-cqkmj"]
Oct 30 21:09:58.049592 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.049562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.049733 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.049611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.049733 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.049643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.049733 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.049663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.049733 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.049680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.049733 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.049699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.049733 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.049725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt2bb\" (UniqueName: \"kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.049953 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.049743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.091804 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.091760    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:09:58.150941 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.150913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.151075 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.150956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.151075 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.150979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.151075 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.151011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.151075 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.151051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2bb\" (UniqueName: \"kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.151273 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.151078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.151273 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.151108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.151273 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.151166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.151918 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.151893    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.152496 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.152461    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.152731 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.152710    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.152884 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.152861    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.153509 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.153483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.156305 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.156280    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.156391 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.156331    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.182553 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.182528    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2bb\" (UniqueName: \"kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.312075 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.311827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:09:58.625232 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.625200    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5fd76869-cqkmj"]
Oct 30 21:09:58.630672 ip-10-0-129-84 kubenswrapper[2581]: W1030 21:09:58.630618    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6dad26_b908_4f23_9948_4dc9636e44f5.slice/crio-73936e054d924a4fd3d155caad80a27fe1fb3ab05035278d14cc9de33a781485 WatchSource:0}: Error finding container 73936e054d924a4fd3d155caad80a27fe1fb3ab05035278d14cc9de33a781485: Status 404 returned error can't find the container with id 73936e054d924a4fd3d155caad80a27fe1fb3ab05035278d14cc9de33a781485
Oct 30 21:09:58.900434 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.900392    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5fd76869-cqkmj" event={"ID":"8f6dad26-b908-4f23-9948-4dc9636e44f5","Type":"ContainerStarted","Data":"9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4"}
Oct 30 21:09:58.900592 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.900446    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5fd76869-cqkmj" event={"ID":"8f6dad26-b908-4f23-9948-4dc9636e44f5","Type":"ContainerStarted","Data":"73936e054d924a4fd3d155caad80a27fe1fb3ab05035278d14cc9de33a781485"}
Oct 30 21:09:58.905505 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.904678    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerStarted","Data":"967ba2f547ef89c5fbadfecec7e766669881d474c6ceaee98dd344ac994caa9f"}
Oct 30 21:09:58.905505 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.904747    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerStarted","Data":"2910049e8b57a7c449466374993c815651f2b60866a41d5d25f42dca69bb05f8"}
Oct 30 21:09:58.905505 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.904785    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerStarted","Data":"7310404d7a12d4e9d0ca929f31096bae89d46797ebf1c5419986e221eaba86e9"}
Oct 30 21:09:58.905505 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.904819    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerStarted","Data":"76add9a95345cb5a0ce8a87ac880a7f6dc82385edc6dba95a38830082144ca05"}
Oct 30 21:09:58.906711 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.906676    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vts2m" event={"ID":"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d","Type":"ContainerStarted","Data":"8658195574c7f0073a41a694cef28a18b289938a659646665bd67895f07faef8"}
Oct 30 21:09:58.906823 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.906800    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:09:58.944148 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.944096    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b5fd76869-cqkmj" podStartSLOduration=1.944079683 podStartE2EDuration="1.944079683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:58.94288772 +0000 UTC m=+70.941347891" watchObservedRunningTime="2026-10-30 21:09:58.944079683 +0000 UTC m=+70.942539841"
Oct 30 21:09:58.993603 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:58.993539    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.817573603 podStartE2EDuration="9.993521637s" totalImagesPullingTime="7.154313447s" totalInitContainerRuntime="21.634587ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:58.992186823 +0000 UTC m=+70.990647003" watchObservedRunningTime="2026-10-30 21:09:58.993521637 +0000 UTC m=+70.991981797"
Oct 30 21:09:59.043535 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:59.043471    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vts2m" podStartSLOduration=65.965864253 podStartE2EDuration="1m10.043455209s" totalImagesPullingTime="4.077590956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:59.042192657 +0000 UTC m=+71.040652815" watchObservedRunningTime="2026-10-30 21:09:59.043455209 +0000 UTC m=+71.041915367"
Oct 30 21:09:59.743169 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:59.743114    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:09:59.799227 ip-10-0-129-84 kubenswrapper[2581]: I1030 21:09:59.799055    2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:09:59.799266 ip-10-0-129-84 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:09:59.821678 ip-10-0-129-84 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:09:59.821925 ip-10-0-129-84 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:09:59.821993 ip-10-0-129-84 systemd[1]: kubelet.service: Consumed 3.986s CPU time.
Oct 30 21:09:59.840820 ip-10-0-129-84 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:09:59.901302 ip-10-0-129-84 kubenswrapper[5886]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:09:59.901302 ip-10-0-129-84 kubenswrapper[5886]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:09:59.901302 ip-10-0-129-84 kubenswrapper[5886]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:09:59.901302 ip-10-0-129-84 kubenswrapper[5886]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:09:59.903271 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903196    5886 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:59.903271 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903271    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:59.903271 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903275    5886 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903279    5886 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903282    5886 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903284    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903287    5886 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903290    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903292    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903295    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903298    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903300    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903303    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903306    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903308    5886 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903311    5886 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903313    5886 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903316    5886 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903319    5886 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903321    5886 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903324    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903326    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:59.903372 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903339    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903342    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903345    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903347    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903350    5886 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903353    5886 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903356    5886 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903358    5886 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903361    5886 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903363    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903366    5886 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903369    5886 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903372    5886 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903375    5886 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903378    5886 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903380    5886 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903383    5886 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903386    5886 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903388    5886 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:59.903856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903391    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903394    5886 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903396    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903399    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903401    5886 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903404    5886 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903406    5886 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903409    5886 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903411    5886 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903416    5886 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903421    5886 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903424    5886 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903427    5886 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903430    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903433    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903436    5886 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903438    5886 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903441    5886 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903444    5886 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:59.904344 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903447    5886 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903450    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903452    5886 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903455    5886 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903458    5886 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903461    5886 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903464    5886 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903466    5886 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903469    5886 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903472    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903474    5886 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903477    5886 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903480    5886 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903484    5886 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903487    5886 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903489    5886 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903492    5886 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903494    5886 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903497    5886 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:59.904854 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903500    5886 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903503    5886 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903506    5886 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903509    5886 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903512    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903514    5886 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903517    5886 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903520    5886 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903522    5886 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903525    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903528    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903531    5886 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903533    5886 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903536    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903539    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903541    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903544    5886 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903547    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903550    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903553    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:59.905328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903556    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903558    5886 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903561    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903564    5886 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903569    5886 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903572    5886 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.903576    5886 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904310    5886 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904317    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904321    5886 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904324    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904327    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904332    5886 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904336    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904340    5886 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904344    5886 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904347    5886 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904350    5886 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904354    5886 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:59.905813 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904357    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904359    5886 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904362    5886 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904365    5886 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904368    5886 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904370    5886 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904373    5886 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904376    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904379    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904382    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904384    5886 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904387    5886 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904390    5886 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904392    5886 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904395    5886 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904398    5886 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904401    5886 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904403    5886 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904406    5886 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:59.906307 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904410    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904414    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904421    5886 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904424    5886 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904427    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904429    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904432    5886 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904435    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904437    5886 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904440    5886 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904442    5886 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904445    5886 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904448    5886 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904450    5886 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904453    5886 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904455    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904458    5886 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904460    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904463    5886 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:59.906782 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904466    5886 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904468    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904471    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904474    5886 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904477    5886 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904479    5886 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904482    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904485    5886 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904488    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904490    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904493    5886 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904496    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904498    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904501    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904504    5886 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904507    5886 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904509    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904512    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904515    5886 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904517    5886 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:59.907275 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904520    5886 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904522    5886 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904525    5886 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904527    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904530    5886 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904532    5886 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904535    5886 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904537    5886 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904540    5886 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904544    5886 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904548    5886 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904551    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904555    5886 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904557    5886 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904560    5886 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904563    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904565    5886 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904568    5886 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904570    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:59.907771 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904573    5886 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904576    5886 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904578    5886 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904581    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904583    5886 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904587    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904589    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904592    5886 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904594    5886 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904597    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904600    5886 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904602    5886 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904604    5886 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904607    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904609    5886 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904612    5886 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.904614    5886 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905051    5886 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905056    5886 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905059    5886 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:59.908263 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905061    5886 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905064    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905067    5886 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905070    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905072    5886 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905075    5886 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905078    5886 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905081    5886 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905083    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905086    5886 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905089    5886 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905092    5886 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905094    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905097    5886 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905099    5886 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905102    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905105    5886 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905107    5886 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905110    5886 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:59.908792 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905112    5886 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905115    5886 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905118    5886 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905120    5886 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905123    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905125    5886 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905128    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905130    5886 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905133    5886 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905136    5886 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905138    5886 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905141    5886 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905143    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905146    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905148    5886 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905165    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905169    5886 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905173    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905176    5886 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:59.909301 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905183    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905185    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905188    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905191    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905194    5886 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905198    5886 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905202    5886 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905205    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905209    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905212    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905215    5886 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905218    5886 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905220    5886 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905223    5886 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905225    5886 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905228    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905231    5886 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905234    5886 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905236    5886 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:59.909772 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905239    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905242    5886 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905244    5886 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905247    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905250    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905253    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905255    5886 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905258    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905261    5886 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905264    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905266    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905269    5886 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905272    5886 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905277    5886 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905279    5886 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905282    5886 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905285    5886 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905287    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905290    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:59.910252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905293    5886 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905295    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905298    5886 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905301    5886 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905303    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905305    5886 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905308    5886 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905310    5886 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905313    5886 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905316    5886 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905318    5886 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905321    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905324    5886 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905328    5886 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905331    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905333    5886 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905336    5886 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905339    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905342    5886 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905344    5886 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:59.910866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905347    5886 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905349    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905353    5886 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905356    5886 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905358    5886 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905361    5886 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.905365    5886 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905461    5886 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905478    5886 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905486    5886 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905492    5886 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905498    5886 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905501    5886 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905506    5886 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905510    5886 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905513    5886 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905516    5886 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905520    5886 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905523    5886 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905526    5886 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905530    5886 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905533    5886 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905536    5886 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:09:59.911720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905539    5886 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905542    5886 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905549    5886 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905552    5886 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905555    5886 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905558    5886 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905561    5886 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905565    5886 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905568    5886 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905571    5886 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905574    5886 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905577    5886 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905580    5886 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905583    5886 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905587    5886 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905591    5886 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905595    5886 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905599    5886 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905602    5886 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905605    5886 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905608    5886 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905611    5886 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905615    5886 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905618    5886 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:09:59.912698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905621    5886 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905624    5886 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905627    5886 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905631    5886 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905634    5886 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905637    5886 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905640    5886 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905644    5886 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905646    5886 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905649    5886 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905654    5886 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905658    5886 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905660    5886 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905663    5886 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905667    5886 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905670    5886 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905674    5886 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905677    5886 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905680    5886 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905683    5886 flags.go:64] FLAG: --help="false"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905686    5886 flags.go:64] FLAG: --hostname-override="ip-10-0-129-84.ec2.internal"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905689    5886 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905693    5886 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:09:59.913353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905698    5886 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905701    5886 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905704    5886 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905707    5886 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905710    5886 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905713    5886 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905716    5886 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905719    5886 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905722    5886 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905725    5886 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905728    5886 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905731    5886 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905734    5886 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905737    5886 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905740    5886 flags.go:64] FLAG: --lock-file=""
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905743    5886 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905746    5886 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905749    5886 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905755    5886 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905759    5886 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905763    5886 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905766    5886 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905769    5886 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905772    5886 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:09:59.913924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905775    5886 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905778    5886 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905782    5886 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905785    5886 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905789    5886 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905792    5886 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905796    5886 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905798    5886 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905803    5886 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905806    5886 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905809    5886 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905812    5886 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905819    5886 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905822    5886 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905826    5886 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905829    5886 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905832    5886 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905835    5886 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905838    5886 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905841    5886 flags.go:64] FLAG: --port="10250"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905844    5886 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905847    5886 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-013d510f19a92b96b"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905851    5886 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905854    5886 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905857    5886 flags.go:64] FLAG: --register-node="true"
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905860    5886 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:09:59.914601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905864    5886 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905868    5886 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905871    5886 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905874    5886 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905878    5886 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905881    5886 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905884    5886 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905887    5886 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905890    5886 flags.go:64] FLAG: --runonce="false"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905893    5886 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905896    5886 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905899    5886 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905902    5886 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905905    5886 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905909    5886 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905912    5886 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905915    5886 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905918    5886 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905921    5886 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905924    5886 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905927    5886 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905930    5886 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905933    5886 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905936    5886 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905941    5886 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:09:59.915239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905944    5886 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905947    5886 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905952    5886 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905954    5886 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905957    5886 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905960    5886 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905963    5886 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905966    5886 flags.go:64] FLAG: --v="2"
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905972    5886 flags.go:64] FLAG: --version="false"
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905976    5886 flags.go:64] FLAG: --vmodule=""
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905981    5886 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.905984    5886 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906111    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906115    5886 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906118    5886 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906121    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906124    5886 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906126    5886 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906129    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906132    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906134    5886 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906138    5886 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:59.916055 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906141    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906144    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906146    5886 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906149    5886 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906164    5886 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906167    5886 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906170    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906173    5886 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906175    5886 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906178    5886 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906181    5886 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906184    5886 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906186    5886 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906189    5886 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906191    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906194    5886 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906197    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906199    5886 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906204    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:59.916815 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906206    5886 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906211    5886 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906215    5886 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906218    5886 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906222    5886 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906225    5886 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906227    5886 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906230    5886 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906233    5886 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906237    5886 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906240    5886 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906243    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906248    5886 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906250    5886 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906253    5886 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906256    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906258    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906261    5886 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906264    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:59.917557 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906266    5886 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906269    5886 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906272    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906274    5886 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906277    5886 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906280    5886 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906283    5886 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906285    5886 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906288    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906290    5886 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906293    5886 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906296    5886 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906300    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906302    5886 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906305    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906308    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906310    5886 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906314    5886 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906317    5886 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906320    5886 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:59.918027 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906323    5886 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906325    5886 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906328    5886 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906331    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906333    5886 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906338    5886 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906340    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906343    5886 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906345    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906348    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906351    5886 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906353    5886 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906355    5886 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906358    5886 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906361    5886 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906364    5886 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906366    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906369    5886 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906371    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906374    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:59.918628 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906377    5886 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906379    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906382    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906384    5886 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906388    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906390    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906393    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906396    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906398    5886 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906401    5886 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906404    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906406    5886 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906409    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906411    5886 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906414    5886 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906416    5886 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906419    5886 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:59.919262 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906423    5886 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.906449    5886 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906484    5886 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906489    5886 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906492    5886 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906495    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906498    5886 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906500    5886 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906503    5886 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906506    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906509    5886 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906512    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906514    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906517    5886 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906520    5886 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906522    5886 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:59.919735 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906525    5886 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906527    5886 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906530    5886 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906532    5886 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906535    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906538    5886 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906540    5886 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906543    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906545    5886 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906548    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906551    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906553    5886 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906556    5886 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906558    5886 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906561    5886 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906570    5886 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906576    5886 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906578    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906581    5886 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906584    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:59.920135 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906586    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906589    5886 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906592    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906594    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906597    5886 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906599    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906602    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906605    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906607    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906610    5886 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906613    5886 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906615    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906618    5886 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906621    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906623    5886 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906626    5886 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906628    5886 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906631    5886 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906634    5886 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:59.920794 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906636    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906639    5886 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906641    5886 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906644    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906647    5886 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906650    5886 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906652    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906655    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906658    5886 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906662    5886 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906665    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906668    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906670    5886 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906673    5886 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906676    5886 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906680    5886 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906683    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906685    5886 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906688    5886 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906690    5886 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:59.921277 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906694    5886 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906696    5886 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906699    5886 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906701    5886 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906704    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906706    5886 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906709    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906711    5886 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906714    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906717    5886 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906721    5886 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906725    5886 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906729    5886 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906733    5886 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906735    5886 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906738    5886 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906741    5886 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906744    5886 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906748    5886 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:59.921785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906751    5886 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906754    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906760    5886 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906762    5886 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906765    5886 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906768    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906770    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906773    5886 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906776    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906778    5886 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906781    5886 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906783    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906786    5886 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.906789    5886 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.910250    5886 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.910266    5886 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.910284    5886 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.910288    5886 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910340    5886 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:59.922267 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910345    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910349    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910353    5886 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910356    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910359    5886 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910362    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910366    5886 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910369    5886 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910372    5886 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910375    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910378    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910381    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910383    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910386    5886 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910389    5886 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910392    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910395    5886 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910398    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910401    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:59.922745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910403    5886 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910406    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910409    5886 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910412    5886 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910415    5886 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910417    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910420    5886 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910422    5886 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910425    5886 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910428    5886 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910431    5886 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910434    5886 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910436    5886 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910439    5886 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910441    5886 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910444    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910446    5886 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910450    5886 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910454    5886 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910457    5886 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:59.923222 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910460    5886 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910463    5886 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910465    5886 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910468    5886 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910470    5886 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910473    5886 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910477    5886 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910479    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910482    5886 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910485    5886 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910487    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910490    5886 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910493    5886 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910496    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910498    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910501    5886 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910503    5886 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910506    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910508    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:59.923719 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910511    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910513    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910516    5886 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910519    5886 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910521    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910524    5886 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910526    5886 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910529    5886 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910531    5886 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910534    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910536    5886 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910539    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910542    5886 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910544    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910547    5886 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910549    5886 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910552    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910555    5886 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910557    5886 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910560    5886 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:59.924214 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910563    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910573    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910576    5886 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910579    5886 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910581    5886 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910584    5886 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910587    5886 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910589    5886 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910592    5886 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910594    5886 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910597    5886 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910599    5886 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910602    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910605    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910608    5886 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910610    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910613    5886 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910616    5886 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910619    5886 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:59.924855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910621    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910625    5886 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910628    5886 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910631    5886 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910633    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910636    5886 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910638    5886 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.910641    5886 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.910668    5886 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911403    5886 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911418    5886 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911423    5886 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911428    5886 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911432    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911437    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911441    5886 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:59.925533 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911452    5886 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911458    5886 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911462    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911467    5886 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911471    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911475    5886 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911479    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911484    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911488    5886 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911493    5886 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911497    5886 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911502    5886 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911511    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911516    5886 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911528    5886 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911533    5886 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911537    5886 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911541    5886 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911545    5886 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911550    5886 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:59.926074 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911555    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911559    5886 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911563    5886 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911568    5886 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911572    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911581    5886 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911585    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911590    5886 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911594    5886 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911598    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911603    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911607    5886 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911611    5886 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911615    5886 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911620    5886 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911624    5886 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911629    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911633    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911642    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911646    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:59.926935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911651    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911655    5886 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911659    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911664    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911668    5886 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911672    5886 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911679    5886 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911687    5886 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911694    5886 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911698    5886 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911713    5886 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911719    5886 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911724    5886 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911729    5886 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911734    5886 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911739    5886 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911743    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911747    5886 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:59.927730 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911752    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911756    5886 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911760    5886 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911765    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911769    5886 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911778    5886 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911783    5886 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911788    5886 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911792    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911796    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911800    5886 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911804    5886 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911809    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911814    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911818    5886 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911822    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911827    5886 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911832    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911842    5886 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:59.928365 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911846    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911851    5886 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911855    5886 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911861    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911865    5886 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911871    5886 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911875    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911879    5886 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911884    5886 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911888    5886 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911893    5886 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911902    5886 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911906    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911911    5886 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911915    5886 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911920    5886 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911924    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911929    5886 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911933    5886 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911937    5886 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:59.928824 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911942    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.911947    5886 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.912003    5886 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912146    5886 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912175    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912182    5886 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912188    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912193    5886 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912198    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912202    5886 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912206    5886 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912217    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912222    5886 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912226    5886 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912231    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912235    5886 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:09:59.929328 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912240    5886 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912244    5886 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912248    5886 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912254    5886 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912259    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912264    5886 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912268    5886 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912278    5886 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912282    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912287    5886 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912292    5886 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912297    5886 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912301    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912307    5886 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912311    5886 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912315    5886 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912320    5886 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912324    5886 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912328    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912339    5886 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:09:59.929706 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912343    5886 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912348    5886 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912354    5886 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912358    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912363    5886 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912367    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912371    5886 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912377    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912390    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912394    5886 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912399    5886 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912440    5886 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912565    5886 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912575    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912579    5886 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912583    5886 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912586    5886 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912591    5886 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912593    5886 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912596    5886 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:09:59.930207 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912600    5886 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912603    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912606    5886 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912609    5886 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912611    5886 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912614    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912618    5886 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912620    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912623    5886 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912626    5886 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912629    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912632    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912635    5886 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912637    5886 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912640    5886 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912643    5886 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912646    5886 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912649    5886 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912651    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912654    5886 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:09:59.930785 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912657    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912660    5886 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912662    5886 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912665    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912668    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912671    5886 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912673    5886 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912678    5886 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912684    5886 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912687    5886 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912690    5886 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912694    5886 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912698    5886 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912701    5886 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912704    5886 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912707    5886 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912710    5886 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912713    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:09:59.931356 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912716    5886 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912719    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912722    5886 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912725    5886 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912728    5886 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912730    5886 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912733    5886 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912738    5886 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912741    5886 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912744    5886 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912746    5886 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912749    5886 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912751    5886 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912754    5886 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:09:59.912757    5886 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.912846    5886 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.913801    5886 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.913860    5886 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.914031    5886 server.go:1014] "Starting client certificate rotation"
Oct 30 21:09:59.931839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.914176    5886 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:09:59.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.914213    5886 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:03:49 +0000 UTC" deadline="2028-06-11 19:01:31.774304092 +0000 UTC"
Oct 30 21:09:59.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.914239    5886 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14157h51m31.86006667s"
Oct 30 21:09:59.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.914478    5886 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:09:59.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.915529    5886 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:09:59.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.916669    5886 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:09:59.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.918521    5886 log.go:25] "Validated CRI v1 image API"
Oct 30 21:09:59.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.919294    5886 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:09:59.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.921191    5886 fs.go:135] Filesystem UUIDs: map[216814b9-f443-4d64-97e0-a5454c6381d8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8678182a-faa6-41bb-a551-4c87234e0796:/dev/nvme0n1p4]
Oct 30 21:09:59.933086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.921213    5886 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0296c1ed3f694373535a6894beabe196ef70f25f4d72eb1b3b1cee149257f31a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0296c1ed3f694373535a6894beabe196ef70f25f4d72eb1b3b1cee149257f31a/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16ef3d349fba818afd960df5f45e40bae40443b675c13a78a763c84bd568cd0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16ef3d349fba818afd960df5f45e40bae40443b675c13a78a763c84bd568cd0a/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f90456bf0363bc50436136b94f582c8a041d6753b55b1398824569283d64a23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f90456bf0363bc50436136b94f582c8a041d6753b55b1398824569283d64a23/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28f1fd56fe18a3f82bcfb1ddf124759362108a7b5bc88d6848b1ddb927d9fd73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28f1fd56fe18a3f82bcfb1ddf124759362108a7b5bc88d6848b1ddb927d9fd73/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/382d88ffbbd482aa8f780935f1b7c598c22babf4fd49491a4b55293069eb58ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/382d88ffbbd482aa8f780935f1b7c598c22babf4fd49491a4b55293069eb58ea/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48e104e518efb95b36d01295ccfad477854de2eed39c12f3ef19b9a056ca5034/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48e104e518efb95b36d01295ccfad477854de2eed39c12f3ef19b9a056ca5034/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d916d909e3acfe19ef7a30c62362a9c2b709d895a44dafb55c4d9f20a76dd2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d916d909e3acfe19ef7a30c62362a9c2b709d895a44dafb55c4d9f20a76dd2c/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52be8e7c3a65736e0349b24059a842cee919e3da1a15b4535772212373c2bc5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52be8e7c3a65736e0349b24059a842cee919e3da1a15b4535772212373c2bc5c/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55aec019a97ffa3ba003eeb7e13ec1d88c580593e50103bfa75902174852842b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55aec019a97ffa3ba003eeb7e13ec1d88c580593e50103bfa75902174852842b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72802bc74c71245f9e1cbbc7a0e887e28a22016d3d23f0eb9baa94c28341d963/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72802bc74c71245f9e1cbbc7a0e887e28a22016d3d23f0eb9baa94c28341d963/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73936e054d924a4fd3d155caad80a27fe1fb3ab05035278d14cc9de33a781485/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73936e054d924a4fd3d155caad80a27fe1fb3ab05035278d14cc9de33a781485/userdata/shm major:0 minor:391 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88fa3b9e3541fe32c2515029381b4b437c267d40f62c8259fa033928ff6aea45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88fa3b9e3541fe32c2515029381b4b437c267d40f62c8259fa033928ff6aea45/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f7620bb05d05c86c2409008afa20f424523fb58062ded4e213f0f4a26f453b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f7620bb05d05c86c2409008afa20f424523fb58062ded4e213f0f4a26f453b7/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa1c07082a8c08eea86e6a8bb896af60fde3d693bfc8d697c63ced91b62d4367/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa1c07082a8c08eea86e6a8bb896af60fde3d693bfc8d697c63ced91b62d4367/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5a59d229cc72aac1bc71a9500eb029a2de65c1d1087b44cce77d6d6c751d9df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5a59d229cc72aac1bc71a9500eb029a2de65c1d1087b44cce77d6d6c751d9df/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8ba03cf6fc6eeae979fa03c374a2f74ea3312c94d40587e7db9b5bf1c28d8b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8ba03cf6fc6eeae979fa03c374a2f74ea3312c94d40587e7db9b5bf1c28d8b1/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba635a1904a15e7f23e2995d8b78409c7edc657f8a5251045993696ce359aa4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba635a1904a15e7f23e2995d8b78409c7edc657f8a5251045993696ce359aa4a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb9c7d5488769aa958b70d25c261ce711c886f6895fb7c9ca0cba97c70f3a17b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb9c7d5488769aa958b70d25c261ce711c886f6895fb7c9ca0cba97c70f3a17b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfab3c5fa3eaa15d46d4a0b62e7bcde86183e3efd58615c9fcc6e2ecb96face9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfab3c5fa3eaa15d46d4a0b62e7bcde86183e3efd58615c9fcc6e2ecb96face9/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcdc41b727c7c4d7fd94605a06c0dff5728d2ddf644c17cc2927894750769af2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcdc41b727c7c4d7fd94605a06c0dff5728d2ddf644c17cc2927894750769af2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9f44828ea5416868e085ae70cc40b9a944612b2654d11fd3b5a2d4337c75b80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9f44828ea5416868e085ae70cc40b9a944612b2654d11fd3b5a2d4337c75b80/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f03898478d98a930901a9bf7be7f94d6b43c50d3cdb3dc3a31a90613fef0e1d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f03898478d98a930901a9bf7be7f94d6b43c50d3cdb3dc3a31a90613fef0e1d6/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/414cbb04-b4d4-4840-911b-353f24025f02/volumes/kubernetes.io~projected/kube-api-access-z49dl:{mountpoint:/var/lib/kubelet/pods/414cbb04-b4d4-4840-911b-353f24025f02/volumes/kubernetes.io~projected/kube-api-access-z49dl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/414cbb04-b4d4-4840-911b-353f24025f02/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/414cbb04-b4d4-4840-911b-353f24025f02/volumes/kubernetes.io~secret/metrics-certs major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42109964-a1d9-479b-b626-76b60f9f49da/volumes/kubernetes.io~projected/kube-api-access-8gmxj:{mountpoint:/var/lib/kubelet/pods/42109964-a1d9-479b-b626-76b60f9f49da/volumes/kubernetes.io~projected/kube-api-access-8gmxj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cb66043-7c25-465f-95e4-3535cb4f35ff/volumes/kubernetes.io~projected/kube-api-access-lqj4h:{mountpoint:/var/lib/kubelet/pods/4cb66043-7c25-465f-95e4-3535cb4f35ff/volumes/kubernetes.io~projected/kube-api-access-lqj4h major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dbdcb7d-b158-484a-832b-30fa8cc62c53/volumes/kubernetes.io~projected/kube-api-access-m28qc:{mountpoint:/var/lib/kubelet/pods/4dbdcb7d-b158-484a-832b-30fa8cc62c53/volumes/kubernetes.io~projected/kube-api-access-m28qc major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dbdcb7d-b158-484a-832b-30fa8cc62c53/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4dbdcb7d-b158-484a-832b-30fa8cc62c53/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6427a41a-4ada-4a89-a06a-b28b84815333/volumes/kubernetes.io~projected/kube-api-access-686tm:{mountpoint:/var/lib/kubelet/pods/6427a41a-4ada-4a89-a06a-b28b84815333/volumes/kubernetes.io~projected/kube-api-access-686tm major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6427a41a-4ada-4a89-a06a-b28b84815333/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/6427a41a-4ada-4a89-a06a-b28b84815333/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~projected/kube-api-access-rgkv7:{mountpoint:/var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~projected/kube-api-access-rgkv7 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d/volumes/kubernetes.io~projected/kube-api-access-n8msz:{mountpoint:/var/lib/kubelet/pods/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d/volumes/kubernetes.io~projected/kube-api-access-n8msz major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71f1519e-b03d-4ee5-9abd-b647e628531b/volumes/kubernetes.io~projected/kube-api-access-fhd95:{mountpoint:/var/lib/kubelet/pods/71f1519e-b03d-4ee5-9abd-b647e628531b/volumes/kubernetes.io~projected/kube-api-access-fhd95 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e7da34c-23be-4c24-977d-8ef4b007fe8f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7e7da34c-23be-4c24-977d-8ef4b007fe8f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/config-reloader/2 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/prometheus/15 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~empty-dir/config-out major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~projected/kube-api-access-s4lfx:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~projected/kube-api-access-s4lfx major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~projected/tls-assets major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/config major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/web-config major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~projected/kube-api-access-mt2bb:{mountpoint:/var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~projected/kube-api-access-mt2bb major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9193a5a4-d86a-47f5-9db2-c23851803a36/volumes/kubernetes.io~projected/kube-api-access-8n8rb:{mountpoint:/var/lib/kubelet/pods/9193a5a4-d86a-47f5-9db2-c23851803a36/volumes/kubernetes.io~projected/kube-api-access-8n8rb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d2612a3-2331-4145-81bc-1c2eea9a37a5/volumes/kubernetes.io~projected/kube-api-access-p6drm:{mountpoint:/var/lib/kubelet/pods/9d2612a3-2331-4145-81bc-1c2eea9a37a5/volumes/kubernetes.io~projected/kube-api-access-p6drm major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d2612a3-2331-4145-81bc-1c2eea9a37a5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9d2612a3-2331-4145-81bc-1c2eea9a37a5/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3c22004-d04f-4ac5-ae97-224bf22bcf0e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a3c22004-d04f-4ac5-ae97-224bf22bcf0e/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~projected/kube-api-access-44gpq:{mountpoint:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~projected/kube-api-access-44gpq major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volumes/kubernetes.io~projected/kube-api-access-pmjf8:{mountpoint:/var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volumes/kubernetes.io~projected/kube-api-access-pmjf8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~projected/kube-api-access-s9xhq:{mountpoint:/var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~projected/kube-api-access-s9xhq major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~projected/kube-api-access-q5b4z:{mountpoint:/var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~projected/kube-api-access-q5b4z major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6d483a0-47e4-49a3-a59e-7e4fe09c8779/volumes/kubernetes.io~projected/kube-api-access-tbnk9:{mountpoint:/var/lib/kubelet/pods/d6d483a0-47e4-49a3-a59e-7e4fe09c8779/volumes/kubernetes.io~projected/kube-api-access-tbnk9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~projected/kube-api-access-mhjfw:{mountpoint:/var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~projected/kube-api-access-mhjfw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffea2206-5878-442e-a1fd-e1a691cfbcbc/volumes/kubernetes.io~projected/kube-api-access-8drfb:{mountpoint:/var/lib/kubelet/pods/ffea2206-5878-442e-a1fd-e1a691cfbcbc/volumes/kubernetes.io~projected/kube-api-access-8drfb major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b9bf822dfe41e40d5f54772ecabf24cbe1a569d99c2bcf6973129ac949cef0e5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a65ad9ff0d223c340a26055ee44db0d1e795941943ea2975fd4e3e4295adbeff/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ae24fade9a90de394d3a929f9faf325664e4d63f473ec3037962ac2dbeebd3d0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/14a48be351b5109bb1e9d618cc68d6483c91d659ea5fbb1af22ca94fabf30f3d/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/acdfc81938a66e351d1fbdd6a432d4378962406d81d60e867caa8dd9f3adf7da/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/03d3f5db0004c34ebf7ed46f67b4a5c6fad3cb0cba356a616b7c64dc3fc46fc9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/be2e93190e04bc508d2a9da26cbad41938cc3e1e2e0901d0030936c9cb40bfe1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d547fdf5a2e3097006db4929d625cfa10e0915a8332c00cee71e93766f69aecd/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ab321c50e55780addee7b2362e61f3706e0a75143efe304bb578f40150f7d3f7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/048cf474a89d3f91f39b55ae937801d6ee3f4b5d5651822cba0694a50e71e6cf/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5c58fdc0701d6a158e2e3ece704f465d129395929ef2f009a25a2ed537be50ff/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/6d157d2f2ec5af398f71460925053b1226431440cf27ab4bc035735e51ecc4c0/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/41c788275fdbce0507d2c8c10cf1aed12bc70e5dce2f242675b3b07856d38314/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b2a8a23dbcf7c73066cd5a681cf1bc3066fdff80b75f1f5dd106826f75be9671/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f6cff9de7f274531721b9c619142dfa3e790a0128ce4f6ac6f2d7fe67e92cbea/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e1516e1c561e5bb4044e68ff8b86e3857d6cf95ee287f84fab3fd019add6b01a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ac31ef883487901655c146e67704a6e61f4e4a1cff66a2f0a6904557fa0f42ca/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7cc2314735f7441ff6b76c2bb3f773077b0c23569776b00cc2a0f5967d6d2a2e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3986ec56546c19d356a067417a5028e9d25e1f187a11220073fa401ed8b55aa1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/34cda7c1c94becd38552a03f38f2532dbf70bc501eef9c96f72fa7506551e161/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0e4967ec54ef4efb8833f40a4347a39c3060f1ab992c0a16bde9f5fafba979ba/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/b0513d376f2d289d1278a79c8b92154e8d9ba31a2354c3af49a88e0f92771cef/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/583ec48b2ac784b372d2b639c0b3c2bc4fbec235678bc76fa2d83a2652fa98c5/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/19ac5cdc81df60d4625ad4fd2bd6e584ecb9fd1069e49ff10b66bae3881e03f0/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/08b625e7ffeefbbc8affed111db37881a1b01b928eee4b63f94eda37d9d1ed5a/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/f44e5ff689f9543243c02a4176791874196c6f202db2d1edb8e8be7972ba10b1/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/df2740dcece46429fcae09460da70b400fcdb502ddd8e6fd43dd802b0c12df8c/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/2462b114f9943f3511870740da241e0e56a33575b0ab2169dce2b575b77c398f/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/3b115c4c4b0308e794225229a76ad11d7adfc233ae62de4c03ce5aa1725f4132/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/82b33ee451fbdbba7dbd00d91cf5a52658f6f06a2361d14cb054f18afbd2695c/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/c46c542585b77a69e0d40daeb1a29c4d4226b034a15373accd0df75e35266498/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/004968a2bf69d0b14721c3b491ef8a5e6d95a5e6d4245519b0808bf7e3e1a458/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/6f1be6e5d399c56c2795863f16d6c4614dd9baef995684a48a2036d86c885d48/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/b1eea2b0f0c1e8754b00c12425b8580379e43373262072f1cb45012ba90e5cf5/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/78f5ab2cc3e89f0450bd9ef6bd955d84be4425c886a1a609587dc0b3892c04af/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/f2981d01397aa1c59a5ecbb1d8f2b51ad6322aa207424f6eba572cdd23085000/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/d23bc2cd2e9eae77d3e94404cd0cc60df733397518de6ff8b6c0a57e581fc7c1/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/09bfa1f2a0d550d44dbe3050c1b834828646a04626920af163714d59b76d39e6/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/1026a04eecf22fe63b8da82dac93a326e1218d46446b8c5b407f20df398f98c6/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/e032f4334cf70f25892fa24ed98a0982f9d666ff25f52b0fcfdca82f71f855e6/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/d9459c35f25a1356b86b20d6bd3a828d1c244bf96cfbf8a10dc9fc8e63e72398/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/524ecb8b94b1c6ad49b6b4802d426f34845a8eee0b6f571a86ee63e909f9b650/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/37f75bab11f69a035427c5efc3a4ac48e00ec4a029103104d04873dedc9c869e/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/7bcfa140c8181f06ec293f493de4a44105510adab7212015c6213d4fdcd188cf/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/91c103c676d1e68c76efd1f437774c6b065a726eed05e07f9de2ced3d8347ae8/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a965057ab872459efa5c588e21661b9d57291e8a66ded04d32c7b1cb3d9ec475/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/b800b4946710e51d35ba7f1864c57c0962e5cb81b8961996578a1a7abd1042fb/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/6e87d8a9f57d404a5119096c2ee9da2dfaf7d754429b679a722a6b72f63d783d/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/10a4dd45988a4943c312eaddfcc905a6e0829faf601dd39df67350ac15dc570c/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/484ee6406471a1ffa278cbdf2bdd4fb2b8f0b25bd9be7bb9e9d2bea831aa2dc6/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/fc7f9cd75cde9fa4d0cadb2af780fe4c7b33abf21330b4d6d012e64c0ffecfa9/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/447222d649831f5e371246ae32b0991c4d85989b28992ed2f5de614bea510b71/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/765f06dfef2f99ba02f188633568f31ea7e241de0401e8578bef0841b30db4d2/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/a4bc4f049b467ec26e88b4cdb442491a280b0c4db1af7b646af6309cf1558c3b/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/0b830cac01c33b3a3a749587055db9bac42bc431c9ef56237c05e4a0d149d2e4/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/2db4db61dd7328f54e7234009a24341bda86a0ea8a9f0e76f2df4f0407f8da7d/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/c0bd923e02f5ee6ff51e74b32773a5661ca6f59953d01f3a6ea7e5b1930ba997/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/d0d136ce6a64396ba052669292b2891d76cd16adc9b0fa85bf9257757e9ea09d/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fa887279ca0f294e6736180eab3e0c4d40245446600e2e013aa7812600ae7006/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9617fbf4eb40eb1d6d3210d9509dd971f32c0d5ed04aebd4ebe020a6b1ba19e2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/eb24be3a64ef82f3d79a68c06ae6ad2395049dc0a39ad2b3427f5a03ed60b291/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2d63211ca30f432d8bd37354085e3d09017679d02a096ffab8d3469c0386edbb/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/1c3daed49d73e398f332b7a12c2ccdf7589feafa2cde3e7d577b068f317e80c5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/8360f3bea3767b615e1a17d502bb771e2cf3ab26430758746cb29bc145f265ba/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/adf682502e172fee6a59cd172613a85ac8e10befcc9e1e10a408c8587d757760/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/9134e5ab46b62d8e456bdfab8a3808231f6a9797a73942f3854544f52814d124/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ef2356ec17cb49d3a027dc92d646023ba4532c4628685cc064e82c7b9e4baf1a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6666a60a6ee0ffd0c66f1c0a8d6549d051a72cae27246a927a30a9b32985f909/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.929486    5886 manager.go:217] Machine: {Timestamp:2026-10-30 21:09:59.929014463 +0000 UTC m=+0.058265910 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099554 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2de583b017580c15ff6a24643a953a SystemUUID:ec2de583-b017-580c-15ff-6a24643a953a BootID:6520064e-a951-4dcb-a649-e0285e1c2d95 Filesystems:[{Device:/var/lib/kubelet/pods/ffea2206-5878-442e-a1fd-e1a691cfbcbc/volumes/kubernetes.io~projected/kube-api-access-8drfb DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71f1519e-b03d-4ee5-9abd-b647e628531b/volumes/kubernetes.io~projected/kube-api-access-fhd95 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d916d909e3acfe19ef7a30c62362a9c2b709d895a44dafb55c4d9f20a76dd2c/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3c22004-d04f-4ac5-ae97-224bf22bcf0e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~projected/kube-api-access-mhjfw DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa1c07082a8c08eea86e6a8bb896af60fde3d693bfc8d697c63ced91b62d4367/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dbdcb7d-b158-484a-832b-30fa8cc62c53/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6d483a0-47e4-49a3-a59e-7e4fe09c8779/volumes/kubernetes.io~projected/kube-api-access-tbnk9 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfab3c5fa3eaa15d46d4a0b62e7bcde86183e3efd58615c9fcc6e2ecb96face9/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73936e054d924a4fd3d155caad80a27fe1fb3ab05035278d14cc9de33a781485/userdata/shm DeviceMajor:0 DeviceMinor:391 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~projected/kube-api-access-s9xhq DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6427a41a-4ada-4a89-a06a-b28b84815333/volumes/kubernetes.io~projected/kube-api-access-686tm DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f03898478d98a930901a9bf7be7f94d6b43c50d3cdb3dc3a31a90613fef0e1d6/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/414cbb04-b4d4-4840-911b-353f24025f02/volumes/kubernetes.io~projected/kube-api-access-z49dl DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~projected/kube-api-access-s4lfx DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48e104e518efb95b36d01295ccfad477854de2eed39c12f3ef19b9a056ca5034/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f842fcd9-3de6-4162-a8f3-217a76df7e50/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9193a5a4-d86a-47f5-9db2-c23851803a36/volumes/kubernetes.io~projected/kube-api-access-8n8rb DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcdc41b727c7c4d7fd94605a06c0dff5728d2ddf644c17cc2927894750769af2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volumes/kubernetes.io~projected/kube-api-access-pmjf8 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16ef3d349fba818afd960df5f45e40bae40443b675c13a78a763c84bd568cd0a/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55aec019a97ffa3ba003eeb7e13ec1d88c580593e50103bfa75902174852842b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d2612a3-2331-4145-81bc-1c2eea9a37a5/volumes/kubernetes.io~projected/kube-api-access-p6drm DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~projected/kube-api-access-rgkv7 DeviceMajor:0 DeviceMinor:248 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28f1fd56fe18a3f82bcfb1ddf124759362108a7b5bc88d6848b1ddb927d9fd73/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6427a41a-4ada-4a89-a06a-b28b84815333/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/414cbb04-b4d4-4840-911b-353f24025f02/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88fa3b9e3541fe32c2515029381b4b437c267d40f62c8259fa033928ff6aea45/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66b15cb4-0694-40c5-aeac-939d1635a93f/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d2612a3-2331-4145-81bc-1c2eea9a37a5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/382d88ffbbd482aa8f780935f1b7c598c22babf4fd49491a4b55293069eb58ea/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba635a1904a15e7f23e2995d8b78409c7edc657f8a5251045993696ce359aa4a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e7da34c-23be-4c24-977d-8ef4b007fe8f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8ba03cf6fc6eeae979fa03c374a2f74ea3312c94d40587e7db9b5bf1c28d8b1/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4dbdcb7d-b158-484a-832b-30fa8cc62c53/volumes/kubernetes.io~projected/kube-api-access-m28qc DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:384 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42109964-a1d9-479b-b626-76b60f9f49da/volumes/kubernetes.io~projected/kube-api-access-8gmxj DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d/volumes/kubernetes.io~projected/kube-api-access-n8msz DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5a59d229cc72aac1bc71a9500eb029a2de65c1d1087b44cce77d6d6c751d9df/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb9c7d5488769aa958b70d25c261ce711c886f6895fb7c9ca0cba97c70f3a17b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52be8e7c3a65736e0349b24059a842cee919e3da1a15b4535772212373c2bc5c/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f90456bf0363bc50436136b94f582c8a041d6753b55b1398824569283d64a23/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f7620bb05d05c86c2409008afa20f424523fb58062ded4e213f0f4a26f453b7/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b14437a8-5976-4417-99fe-481d0c2ea6b4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~projected/kube-api-access-q5b4z DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9f44828ea5416868e085ae70cc40b9a944612b2654d11fd3b5a2d4337c75b80/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4cb66043-7c25-465f-95e4-3535cb4f35ff/volumes/kubernetes.io~projected/kube-api-access-lqj4h DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adf69dd8-b830-4e07-85d0-4950dcd2c9cf/volumes/kubernetes.io~projected/kube-api-access-44gpq DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:383 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72802bc74c71245f9e1cbbc7a0e887e28a22016d3d23f0eb9baa94c28341d963/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0296c1ed3f694373535a6894beabe196ef70f25f4d72eb1b3b1cee149257f31a/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes/kubernetes.io~projected/kube-api-access-mt2bb DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16ef3d349fba818 MacAddress:42:36:42:91:7e:f4 Speed:10000 Mtu:8901} {Name:28f1fd56fe18a3f MacAddress:86:6c:9e:45:35:c0 Speed:10000 Mtu:8901} {Name:382d88ffbbd482a MacAddress:f6:8b:2e:2d:71:5a Speed:10000 Mtu:8901} {Name:48e104e518efb95 MacAddress:be:07:04:92:f1:62 Speed:10000 Mtu:8901} {Name:4d916d909e3acfe MacAddress:a6:89:09:9f:21:4a Speed:10000 Mtu:8901} {Name:52be8e7c3a65736 MacAddress:ea:cf:60:5b:66:ed Speed:10000 Mtu:8901} {Name:73936e054d924a4 MacAddress:ae:dc:18:fa:54:dc Speed:10000 Mtu:8901} {Name:bb9c7d5488769aa MacAddress:de:ad:a0:21:68:95 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ed:07:b8:cb Speed:0 Mtu:9001} {Name:br-int MacAddress:72:93:18:45:8a:81 Speed:0 Mtu:8901} {Name:cfab3c5fa3eaa15 MacAddress:66:cb:22:5c:39:47 Speed:10000 Mtu:8901} {Name:e9f44828ea54168 MacAddress:ae:c3:0b:34:ff:6b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ed:07:b8:cb Speed:0 Mtu:9001} {Name:f03898478d98a93 MacAddress:ee:0e:ee:ed:13:4e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b6:cb:59:2a:9a:47 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:6c:21:da:4a:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.929819    5886 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.929863    5886 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930133    5886 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930178    5886 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930300    5886 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930308    5886 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930322    5886 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930341    5886 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930434    5886 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930590    5886 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930684    5886 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930716    5886 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930725    5886 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930734    5886 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930742    5886 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.930750    5886 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.931523    5886 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.931537    5886 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:09:59.933532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.932000    5886 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:09:59.935371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935353    5886 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:09:59.935371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935369    5886 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:09:59.935526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935486    5886 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:09:59.935753 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935737    5886 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:09:59.935824 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935777    5886 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:09:59.935907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935896    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:09:59.935968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935917    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:09:59.935968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935927    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:09:59.935968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935937    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:09:59.935968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935946    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:09:59.935968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935956    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:09:59.935968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935966    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:09:59.936281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935976    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:09:59.936281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935988    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:09:59.936281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.935997    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:09:59.936281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.936016    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:09:59.936281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.936032    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:09:59.936281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.936076    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:09:59.936281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.936086    5886 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:09:59.939545 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.939528    5886 server.go:1263] "Started kubelet"
Oct 30 21:09:59.939651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.939579    5886 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:09:59.939700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.939624    5886 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:09:59.939740 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.939699    5886 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:09:59.940043 ip-10-0-129-84 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:09:59.940207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.940182    5886 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:09:59.941424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.941382    5886 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:09:59.948055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.948035    5886 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:09:59.948182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.948053    5886 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:09:59.948182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.948092    5886 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:03:50 +0000 UTC" deadline="2028-04-19 09:21:55.383073773 +0000 UTC"
Oct 30 21:09:59.948182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.948111    5886 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12876h11m55.434966583s"
Oct 30 21:09:59.948182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.948114    5886 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:09:59.948182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.948126    5886 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:09:59.948182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.948173    5886 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:09:59.948469 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:09:59.948279    5886 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.949652    5886 factory.go:153] Registering CRI-O factory
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951228    5886 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.950718    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b15cb4-0694-40c5-aeac-939d1635a93f" volumeName="kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951293    5886 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951290    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b15cb4-0694-40c5-aeac-939d1635a93f" volumeName="kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951304    5886 factory.go:55] Registering systemd factory
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951312    5886 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951314    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951329    5886 factory.go:103] Registering Raw factory
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951347    5886 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951327    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951367    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c35b113e-843a-429a-883c-053ea13d9d0a" volumeName="kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951380    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffea2206-5878-442e-a1fd-e1a691cfbcbc" volumeName="kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951391    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="414cbb04-b4d4-4840-911b-353f24025f02" volumeName="kubernetes.io/projected/414cbb04-b4d4-4840-911b-353f24025f02-kube-api-access-z49dl" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951404    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cb66043-7c25-465f-95e4-3535cb4f35ff" volumeName="kubernetes.io/configmap/4cb66043-7c25-465f-95e4-3535cb4f35ff-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951420    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dbdcb7d-b158-484a-832b-30fa8cc62c53" volumeName="kubernetes.io/empty-dir/4dbdcb7d-b158-484a-832b-30fa8cc62c53-data-volume" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951432    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6427a41a-4ada-4a89-a06a-b28b84815333" volumeName="kubernetes.io/empty-dir/6427a41a-4ada-4a89-a06a-b28b84815333-tmp-dir" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951446    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6dad26-b908-4f23-9948-4dc9636e44f5" volumeName="kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951475    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f842fcd9-3de6-4162-a8f3-217a76df7e50" volumeName="kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-tuned" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951486    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffea2206-5878-442e-a1fd-e1a691cfbcbc" volumeName="kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951499    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951514    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf69dd8-b830-4e07-85d0-4950dcd2c9cf" volumeName="kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951527    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf69dd8-b830-4e07-85d0-4950dcd2c9cf" volumeName="kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951538    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="414cbb04-b4d4-4840-911b-353f24025f02" volumeName="kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951550    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dbdcb7d-b158-484a-832b-30fa8cc62c53" volumeName="kubernetes.io/configmap/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951563    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c22004-d04f-4ac5-ae97-224bf22bcf0e" volumeName="kubernetes.io/configmap/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951575    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf69dd8-b830-4e07-85d0-4950dcd2c9cf" volumeName="kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951586    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e1ba50-dfdf-4627-8ab6-97d9b1598d51" volumeName="kubernetes.io/empty-dir/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-textfile" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951597    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffea2206-5878-442e-a1fd-e1a691cfbcbc" volumeName="kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951609    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6427a41a-4ada-4a89-a06a-b28b84815333" volumeName="kubernetes.io/secret/6427a41a-4ada-4a89-a06a-b28b84815333-metrics-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951620    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b15cb4-0694-40c5-aeac-939d1635a93f" volumeName="kubernetes.io/projected/66b15cb4-0694-40c5-aeac-939d1635a93f-kube-api-access-rgkv7" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951631    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f1519e-b03d-4ee5-9abd-b647e628531b" volumeName="kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951644    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6dad26-b908-4f23-9948-4dc9636e44f5" volumeName="kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951656    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b14437a8-5976-4417-99fe-481d0c2ea6b4" volumeName="kubernetes.io/secret/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951681    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c35b113e-843a-429a-883c-053ea13d9d0a" volumeName="kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951699    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e1ba50-dfdf-4627-8ab6-97d9b1598d51" volumeName="kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951714    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dbdcb7d-b158-484a-832b-30fa8cc62c53" volumeName="kubernetes.io/projected/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-api-access-m28qc" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951726    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951739    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9193a5a4-d86a-47f5-9db2-c23851803a36" volumeName="kubernetes.io/configmap/9193a5a4-d86a-47f5-9db2-c23851803a36-serviceca" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951750    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b14437a8-5976-4417-99fe-481d0c2ea6b4" volumeName="kubernetes.io/projected/b14437a8-5976-4417-99fe-481d0c2ea6b4-kube-api-access-pmjf8" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951762    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e1ba50-dfdf-4627-8ab6-97d9b1598d51" volumeName="kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951775    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6d483a0-47e4-49a3-a59e-7e4fe09c8779" volumeName="kubernetes.io/projected/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kube-api-access-tbnk9" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951787    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f842fcd9-3de6-4162-a8f3-217a76df7e50" volumeName="kubernetes.io/projected/f842fcd9-3de6-4162-a8f3-217a76df7e50-kube-api-access-mhjfw" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951798    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b15cb4-0694-40c5-aeac-939d1635a93f" volumeName="kubernetes.io/configmap/66b15cb4-0694-40c5-aeac-939d1635a93f-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951809    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f1519e-b03d-4ee5-9abd-b647e628531b" volumeName="kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951821    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951832    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951843    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951853    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf69dd8-b830-4e07-85d0-4950dcd2c9cf" volumeName="kubernetes.io/projected/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-kube-api-access-44gpq" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951866    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf69dd8-b830-4e07-85d0-4950dcd2c9cf" volumeName="kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951877    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b14437a8-5976-4417-99fe-481d0c2ea6b4" volumeName="kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-env-overrides" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951889    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cb66043-7c25-465f-95e4-3535cb4f35ff" volumeName="kubernetes.io/projected/4cb66043-7c25-465f-95e4-3535cb4f35ff-kube-api-access-lqj4h" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951900    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6427a41a-4ada-4a89-a06a-b28b84815333" volumeName="kubernetes.io/configmap/6427a41a-4ada-4a89-a06a-b28b84815333-config-volume" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951912    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951923    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951934    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951949    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b14437a8-5976-4417-99fe-481d0c2ea6b4" volumeName="kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951961    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c35b113e-843a-429a-883c-053ea13d9d0a" volumeName="kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951979    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f842fcd9-3de6-4162-a8f3-217a76df7e50" volumeName="kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-tmp" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951991    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952003    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952014    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b14437a8-5976-4417-99fe-481d0c2ea6b4" volumeName="kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952027    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c35b113e-843a-429a-883c-053ea13d9d0a" volumeName="kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952050    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42109964-a1d9-479b-b626-76b60f9f49da" volumeName="kubernetes.io/projected/42109964-a1d9-479b-b626-76b60f9f49da-kube-api-access-8gmxj" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952063    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6427a41a-4ada-4a89-a06a-b28b84815333" volumeName="kubernetes.io/projected/6427a41a-4ada-4a89-a06a-b28b84815333-kube-api-access-686tm" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952075    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6dad26-b908-4f23-9948-4dc9636e44f5" volumeName="kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952087    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6dad26-b908-4f23-9948-4dc9636e44f5" volumeName="kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952099    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e1ba50-dfdf-4627-8ab6-97d9b1598d51" volumeName="kubernetes.io/projected/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-kube-api-access-q5b4z" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952111    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffea2206-5878-442e-a1fd-e1a691cfbcbc" volumeName="kubernetes.io/projected/ffea2206-5878-442e-a1fd-e1a691cfbcbc-kube-api-access-8drfb" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952124    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952136    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952147    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952176    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6dad26-b908-4f23-9948-4dc9636e44f5" volumeName="kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952188    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9193a5a4-d86a-47f5-9db2-c23851803a36" volumeName="kubernetes.io/projected/9193a5a4-d86a-47f5-9db2-c23851803a36-kube-api-access-8n8rb" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952202    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f8e3f58-0a6e-41fa-a599-9d68ae65f01d" volumeName="kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952213    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952224    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c22004-d04f-4ac5-ae97-224bf22bcf0e" volumeName="kubernetes.io/secret/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-agent-certs" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952235    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf69dd8-b830-4e07-85d0-4950dcd2c9cf" volumeName="kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-federate-client-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952246    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e1ba50-dfdf-4627-8ab6-97d9b1598d51" volumeName="kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952257    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6dad26-b908-4f23-9948-4dc9636e44f5" volumeName="kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952270    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952282    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952293    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952304    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6dad26-b908-4f23-9948-4dc9636e44f5" volumeName="kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952316    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf69dd8-b830-4e07-85d0-4950dcd2c9cf" volumeName="kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-client-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952327    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adf69dd8-b830-4e07-85d0-4950dcd2c9cf" volumeName="kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952336    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c35b113e-843a-429a-883c-053ea13d9d0a" volumeName="kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952343    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dbdcb7d-b158-484a-832b-30fa8cc62c53" volumeName="kubernetes.io/secret/4dbdcb7d-b158-484a-832b-30fa8cc62c53-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952350    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71f1519e-b03d-4ee5-9abd-b647e628531b" volumeName="kubernetes.io/projected/71f1519e-b03d-4ee5-9abd-b647e628531b-kube-api-access-fhd95" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952358    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6dad26-b908-4f23-9948-4dc9636e44f5" volumeName="kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952366    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d2612a3-2331-4145-81bc-1c2eea9a37a5" volumeName="kubernetes.io/projected/9d2612a3-2331-4145-81bc-1c2eea9a37a5-kube-api-access-p6drm" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.951704    5886 manager.go:319] Starting recovery of all containers
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952377    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e1ba50-dfdf-4627-8ab6-97d9b1598d51" volumeName="kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952391    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42109964-a1d9-479b-b626-76b60f9f49da" volumeName="kubernetes.io/empty-dir/42109964-a1d9-479b-b626-76b60f9f49da-tmp-dir" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952404    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7da34c-23be-4c24-977d-8ef4b007fe8f" volumeName="kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952415    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d2612a3-2331-4145-81bc-1c2eea9a37a5" volumeName="kubernetes.io/secret/9d2612a3-2331-4145-81bc-1c2eea9a37a5-cert" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952429    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c35b113e-843a-429a-883c-053ea13d9d0a" volumeName="kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952442    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bea9add-f465-49bf-ac0f-c2586e641917" volumeName="kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952453    5886 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c35b113e-843a-429a-883c-053ea13d9d0a" volumeName="kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca" seLinuxMountContext=""
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952466    5886 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:09:59.954049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.952474    5886 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:09:59.978289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.978249    5886 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:09:59.979600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.979576    5886 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:09:59.979600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.979599    5886 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:09:59.980130 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.980084    5886 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:09:59.980258 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:09:59.980143    5886 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:09:59.988387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.988366    5886 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:09:59.988512 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.988393    5886 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:09:59.989427 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.989354    5886 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:09:59.990501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.990465    5886 generic.go:356] "Generic (PLEG): container finished" podID="8bea9add-f465-49bf-ac0f-c2586e641917" containerID="e0702c68f1e9803de7ce4b24d262a10d472f1714a011375fadcce82c7bafa920" exitCode=0
Oct 30 21:09:59.995322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:09:59.995297    5886 generic.go:356] "Generic (PLEG): container finished" podID="d4e1ba50-dfdf-4627-8ab6-97d9b1598d51" containerID="e2aa269852dbc58f8ff39b716369239c3990517bedf8921ecc76c0ea47f9e26d" exitCode=0
Oct 30 21:10:00.000938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.000915    5886 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="494638786be54395d374ed00a927a8c7cc47bac16692a2db8eae9efa8285f240" exitCode=0
Oct 30 21:10:00.000938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.000936    5886 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="91447f050047fe0ba00608b0e5fc28ed1ec37c7594f4531345e3981a06be9c9d" exitCode=0
Oct 30 21:10:00.001059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.000948    5886 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="4b6649b24e275ac2f6c69cab18b8fb8606f818af13682db7e859df02fdae0d72" exitCode=0
Oct 30 21:10:00.001059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.000959    5886 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="affbfc60f1f1599f5f5f809092e0b27a3b0cd809018fb9d604b9aea072b8e8a9" exitCode=0
Oct 30 21:10:00.001059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.000968    5886 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="5bb50dd3dbcf1fd4f382f0c5a96780e7eaf5519e2963240de2bf3b067a66ee67" exitCode=0
Oct 30 21:10:00.001059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.000977    5886 generic.go:356] "Generic (PLEG): container finished" podID="ffea2206-5878-442e-a1fd-e1a691cfbcbc" containerID="2539229ba756edc3548be46b0bfd983b14eecd62c12c2b02a663eb26f70fa170" exitCode=0
Oct 30 21:10:00.010335 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.010317    5886 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:10:00.011293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.011268    5886 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:10:00.030978 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.030952    5886 generic.go:356] "Generic (PLEG): container finished" podID="45708598802090d7fd0a00f30bfad720" containerID="6dac8484e91f1e6311cc42f2a5aa993f9a66a0128171cb4621effc35665c0d6b" exitCode=0
Oct 30 21:10:00.050090 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.050040    5886 manager.go:324] Recovery completed
Oct 30 21:10:00.070896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.070870    5886 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:10:00.070896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.070887    5886 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:10:00.071055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.070906    5886 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:10:00.071110 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071069    5886 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:10:00.071110 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071086    5886 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:10:00.071221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071110    5886 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:10:00.071221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071119    5886 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:10:00.071221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071134    5886 policy_none.go:50] "Start"
Oct 30 21:10:00.071221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071147    5886 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:10:00.071221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071175    5886 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:10:00.071441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071350    5886 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:10:00.071441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071361    5886 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:10:00.071441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.071369    5886 policy_none.go:44] "Start"
Oct 30 21:10:00.075790 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.075763    5886 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:10:00.078429 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.078415    5886 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:10:00.079019 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:00.078439    5886 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:10:00.079019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.079021    5886 server.go:87] "Starting device plugin registration server"
Oct 30 21:10:00.079292 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.079281    5886 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:10:00.079326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.079293    5886 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:10:00.079405 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.079385    5886 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:10:00.079526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.079488    5886 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:10:00.079593 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.079534    5886 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:10:00.079593 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.079541    5886 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:10:00.079682 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.079589    5886 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:10:00.079523565Z","UUID":"09f7a6c0-c5ab-4dc9-88bd-73475a90669e","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:10:00.080552 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.080505    5886 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"]
Oct 30 21:10:00.081930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.081312    5886 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:10:00.081930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.081347    5886 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:10:00.081930 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:00.081721    5886 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:10:00.082720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.082701    5886 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.084394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.083996    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"45708598802090d7fd0a00f30bfad720","Type":"ContainerStarted","Data":"69eea6ceab383a9159339028596231d8e7c6ac8402f8778ff2b80443f684407f"}
Oct 30 21:10:00.084394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.084062    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"45708598802090d7fd0a00f30bfad720","Type":"ContainerDied","Data":"6dac8484e91f1e6311cc42f2a5aa993f9a66a0128171cb4621effc35665c0d6b"}
Oct 30 21:10:00.084394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.084090    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"45708598802090d7fd0a00f30bfad720","Type":"ContainerStarted","Data":"b8ba03cf6fc6eeae979fa03c374a2f74ea3312c94d40587e7db9b5bf1c28d8b1"}
Oct 30 21:10:00.084394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.084111    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" event={"ID":"bd04ddc0cbac70d5dc4161808c80da69","Type":"ContainerStarted","Data":"e411c035255608c635dc7a98f183f95476e42244c48a44e01120f205069402c1"}
Oct 30 21:10:00.084394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.084134    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" event={"ID":"bd04ddc0cbac70d5dc4161808c80da69","Type":"ContainerStarted","Data":"ba635a1904a15e7f23e2995d8b78409c7edc657f8a5251045993696ce359aa4a"}
Oct 30 21:10:00.084394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.084389    5886 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.106320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.106300    5886 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:10:00.106418 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:00.106356    5886 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.106935 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.106921    5886 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:10:00.107007 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:00.106961    5886 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.180413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.180368    5886 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:10:00.182297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.182277    5886 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 21:10:00.182391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.182312    5886 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 21:10:00.182391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.182322    5886 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Oct 30 21:10:00.182391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.182366    5886 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.189479 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.189464    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.189539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.189487    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.189539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.189504    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd04ddc0cbac70d5dc4161808c80da69-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bd04ddc0cbac70d5dc4161808c80da69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.213324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.213305    5886 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.213408 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.213359    5886 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.290173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.290133    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd04ddc0cbac70d5dc4161808c80da69-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bd04ddc0cbac70d5dc4161808c80da69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.290306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.290185    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.290306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.290201    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.290306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.290246    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.290306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.290250    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45708598802090d7fd0a00f30bfad720-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"45708598802090d7fd0a00f30bfad720\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.290306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.290250    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd04ddc0cbac70d5dc4161808c80da69-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"bd04ddc0cbac70d5dc4161808c80da69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:00.934658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.931037    5886 apiserver.go:52] "Watching apiserver"
Oct 30 21:10:00.980316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.980283    5886 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:10:00.980973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.980946    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-n9vj8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal","openshift-multus/multus-w5hw9","openshift-cluster-node-tuning-operator/tuned-99v8c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw","kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal","openshift-multus/network-metrics-daemon-p67fc","openshift-network-diagnostics/network-check-target-vts2m","openshift-dns/node-resolver-tsz8h","openshift-ovn-kubernetes/ovnkube-node-tlfh6","openshift-insights/insights-runtime-extractor-gvjst","openshift-dns/dns-default-zsrxr","openshift-ingress-canary/ingress-canary-krgh5","openshift-monitoring/node-exporter-7bnkf","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/telemeter-client-59b7858fc8-fr72g","openshift-network-operator/iptables-alerter-slp8w","openshift-image-registry/node-ca-q5wbv","openshift-multus/multus-additional-cni-plugins-h2wqw","openshift-console/console-786f578d97-qkqng","openshift-console/console-b5fd76869-cqkmj","openshift-monitoring/prometheus-operator-9cd89b74c-kfdst","kube-system/global-pull-secret-syncer-kmwrj"]
Oct 30 21:10:00.996120 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996078    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:10:00.996120 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996097    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:10:00.996120 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996115    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:10:00.996120 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996089    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:10:00.996407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996189    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hscbl\""
Oct 30 21:10:00.996407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996236    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:10:00.996471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996449    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s8m2s\""
Oct 30 21:10:00.996540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996527    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:10:00.996828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996814    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:10:00.996966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.996953    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nxx4w\""
Oct 30 21:10:00.997035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.997023    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:10:00.997596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.997582    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wjvtw\""
Oct 30 21:10:00.998310 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.998296    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:10:00.998385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.998322    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:10:00.998385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.998372    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:10:00.998475 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.998392    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:10:00.998475 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.998423    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b5899\""
Oct 30 21:10:00.998475 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:00.998455    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.005568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.005545    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.005568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.005546    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.005729 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.005620    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.005729 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.005683    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.006734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.006718    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:10:01.006734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.006729    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.006972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.006957    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.007053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.006996    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:10:01.007053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.007031    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.007053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.007031    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:10:01.007183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.007055    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.007183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.007077    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:10:01.007493 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.007480    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.008298 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.008283    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jv27h\""
Oct 30 21:10:01.008378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.008316    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbg6s\""
Oct 30 21:10:01.008378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.008353    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t284x\""
Oct 30 21:10:01.008476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.008384    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:10:01.008476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.008397    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:10:01.008476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.008455    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q2tg5\""
Oct 30 21:10:01.008573 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.008489    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d6fgf\""
Oct 30 21:10:01.008705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.008692    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 21:10:01.009303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009283    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 21:10:01.009387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009335    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w6j9l\""
Oct 30 21:10:01.009455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009406    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:10:01.009507 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009479    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:10:01.009546 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009521    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8rdf8\""
Oct 30 21:10:01.009786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009763    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:10:01.009879 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009809    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sw2nt\""
Oct 30 21:10:01.009879 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009832    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 21:10:01.009879 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009869    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 21:10:01.009992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009962    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.010023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.009998    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:10:01.010023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010005    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 21:10:01.010023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010016    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:10:01.010113 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010063    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:10:01.010113 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010079    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:10:01.010205 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010192    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:10:01.010245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010149    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.010245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010206    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:10:01.010343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010249    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:10:01.010343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010270    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.010343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010280    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6qvgx\""
Oct 30 21:10:01.010343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010280    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:10:01.010343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010281    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.010343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010311    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:10:01.010343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010324    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:10:01.010611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010398    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-48zwb\""
Oct 30 21:10:01.010611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010402    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28pwt\""
Oct 30 21:10:01.010611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010403    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:10:01.010611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010471    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:10:01.010611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.010484    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.021915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.021886    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:10:01.022981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.022960    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 21:10:01.023195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.023183    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:10:01.036169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036134    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:10:01.036262 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036220    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:10:01.036262 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036256    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:10:01.036361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036276    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:10:01.036361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036291    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:10:01.036557 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036536    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hnk74\""
Oct 30 21:10:01.036557 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036553    5886 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:01.036711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036605    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3q4vg4fcnnq04\""
Oct 30 21:10:01.036711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036622    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:10:01.036711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036677    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4xf7\""
Oct 30 21:10:01.036711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036705    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:10:01.036856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036843    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:10:01.036904 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036895    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v2rbx\""
Oct 30 21:10:01.036954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036895    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:10:01.036954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.036941    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:10:01.037020 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.037001    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2bqgz\""
Oct 30 21:10:01.037075 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.037061    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:10:01.050919 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.050898    5886 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:10:01.057410 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.057386    5886 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:10:01.057496 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:01.057438    5886 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Oct 30 21:10:01.089382 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.089355    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786f578d97-qkqng"]
Oct 30 21:10:01.089587 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:01.089563    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[console-config console-oauth-config console-serving-cert kube-api-access-s9xhq oauth-serving-cert service-ca tmp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-console/console-786f578d97-qkqng" podUID="c35b113e-843a-429a-883c-053ea13d9d0a"
Oct 30 21:10:01.093237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093214    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.093323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093247    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-netns\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.093323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093264    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-sys\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.093323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093280    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.093323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093304    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-hostroot\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.093535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093366    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dbdcb7d-b158-484a-832b-30fa8cc62c53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.093535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093462    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.093535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093500    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.093535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093531    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-device-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.093692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093561    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-textfile\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.093692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093586    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-cni-binary-copy\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.093692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093612    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-sys\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.093692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093647    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dbdcb7d-b158-484a-832b-30fa8cc62c53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.093692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093663    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:10:01.093692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093678    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093696    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093720    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093720    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093742    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093768    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686tm\" (UniqueName: \"kubernetes.io/projected/6427a41a-4ada-4a89-a06a-b28b84815333-kube-api-access-686tm\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093900    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093919    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093801    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093932    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-cni-binary-copy\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093948    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66b15cb4-0694-40c5-aeac-939d1635a93f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093776    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-textfile\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.093971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093975    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-etc-kubernetes\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.094456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.093989    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.094456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094003    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8drfb\" (UniqueName: \"kubernetes.io/projected/ffea2206-5878-442e-a1fd-e1a691cfbcbc-kube-api-access-8drfb\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.094456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094039    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-socket-dir-parent\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.094456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094063    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e7da34c-23be-4c24-977d-8ef4b007fe8f-original-pull-secret\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:10:01.094456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094065    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-lib-modules\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.094456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094140    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjfw\" (UniqueName: \"kubernetes.io/projected/f842fcd9-3de6-4162-a8f3-217a76df7e50-kube-api-access-mhjfw\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.094702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094473    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.094702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094494    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.094702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094514    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.094702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094236    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.094702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094535    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9xhq\" (UniqueName: \"kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.094702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094202    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.094702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094669    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cb66043-7c25-465f-95e4-3535cb4f35ff-iptables-alerter-script\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:10:01.094702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094694    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094706    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-wtmp\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094735    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094759    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-node-log\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094826    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094841    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094851    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cb66043-7c25-465f-95e4-3535cb4f35ff-iptables-alerter-script\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094888    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-serving-certs-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094920    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqj4h\" (UniqueName: \"kubernetes.io/projected/4cb66043-7c25-465f-95e4-3535cb4f35ff-kube-api-access-lqj4h\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094943    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-tls\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.095084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.094959    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-netns\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095180    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-system-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095193    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-serving-certs-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095203    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmxj\" (UniqueName: \"kubernetes.io/projected/42109964-a1d9-479b-b626-76b60f9f49da-kube-api-access-8gmxj\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095203    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-tls\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095220    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6drm\" (UniqueName: \"kubernetes.io/projected/9d2612a3-2331-4145-81bc-1c2eea9a37a5-kube-api-access-p6drm\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095249    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-tmp\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095334    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095335    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-tmp\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095456    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.095498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095488    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-conf-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.095760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095672    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.095760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095709    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.095760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095716    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:10:01.095891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095780    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49dl\" (UniqueName: \"kubernetes.io/projected/414cbb04-b4d4-4840-911b-353f24025f02-kube-api-access-z49dl\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:10:01.095891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095853    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-multus\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.096003 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095981    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:10:01.096060 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.095989    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-registration-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.096210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096190    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-agent-certs\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:10:01.096300 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096209    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:10:01.096300 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096238    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.096401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096297    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.096401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096302    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:10:01.096401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096314    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:10:01.096401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096395    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.096591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096443    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-cnibin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.096591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096478    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysconfig\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.096591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096495    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:10:01.096591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096506    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:10:01.096591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096513    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:10:01.096591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096513    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-tuned\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.096591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096558    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-var-lib-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.096591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096565    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-agent-certs\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:10:01.096951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096600    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-tuned\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.096951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096617    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-var-lib-kubelet\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.096951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096658    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnk9\" (UniqueName: \"kubernetes.io/projected/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kube-api-access-tbnk9\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.096951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096768    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-konnectivity-ca\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:10:01.096951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096773    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:10:01.096951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096831    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.096951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096854    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6427a41a-4ada-4a89-a06a-b28b84815333-tmp-dir\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:01.096951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096914    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.096968    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-bin\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097040    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6427a41a-4ada-4a89-a06a-b28b84815333-tmp-dir\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097057    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097041    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cnibin\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097109    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097212    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097310    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-bin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097423    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-socket-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.097519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097478    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097531    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097621    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjf8\" (UniqueName: \"kubernetes.io/projected/b14437a8-5976-4417-99fe-481d0c2ea6b4-kube-api-access-pmjf8\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097730    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097728    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097772    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097794    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097802    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-kubelet\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097923    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44gpq\" (UniqueName: \"kubernetes.io/projected/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-kube-api-access-44gpq\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.097993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.097958    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/66b15cb4-0694-40c5-aeac-939d1635a93f-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098016    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-modprobe-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098108    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098118    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dbdcb7d-b158-484a-832b-30fa8cc62c53-data-volume\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098192    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098267    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dbdcb7d-b158-484a-832b-30fa8cc62c53-data-volume\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098338    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgkv7\" (UniqueName: \"kubernetes.io/projected/66b15cb4-0694-40c5-aeac-939d1635a93f-kube-api-access-rgkv7\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098467    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098514    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098593    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-metrics-client-ca\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.098626 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098621    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098649    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098713    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-multus-certs\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098743    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42109964-a1d9-479b-b626-76b60f9f49da-hosts-file\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098772    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098781    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098800    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6427a41a-4ada-4a89-a06a-b28b84815333-metrics-tls\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098825    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-dbus\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098851    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098877    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098903    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-k8s-cni-cncf-io\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098935    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.098937    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28qc\" (UniqueName: \"kubernetes.io/projected/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-api-access-m28qc\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099014    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.099036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099041    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.099666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099066    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.099666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099111    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2bb\" (UniqueName: \"kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.099666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099188    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-metrics-client-ca\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.099666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099480    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.099666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099503    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:10:01.099666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099515    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.099666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099665    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.100047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099677    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.100047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.099974    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.100284 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100261    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3c22004-d04f-4ac5-ae97-224bf22bcf0e-konnectivity-ca\") pod \"konnectivity-agent-n9vj8\" (UID: \"a3c22004-d04f-4ac5-ae97-224bf22bcf0e\") " pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:10:01.100578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100557    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:10:01.100821 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100789    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-systemd-units\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.100914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100836    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-netd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.100914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100869    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.100914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100901    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-metrics-client-ca\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.101067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100931    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dbdcb7d-b158-484a-832b-30fa8cc62c53-crio-socket\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.101067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100949    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.101067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100957    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9193a5a4-d86a-47f5-9db2-c23851803a36-host\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:10:01.101067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.100983    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.101067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101009    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.101067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101016    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6427a41a-4ada-4a89-a06a-b28b84815333-metrics-tls\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:01.101067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101036    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-etc-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.101067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101063    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-script-lib\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.101450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101076    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.101450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101173    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-metrics-client-ca\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.101450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101150    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d2612a3-2331-4145-81bc-1c2eea9a37a5-cert\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:10:01.101450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101302    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.101450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101305    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-script-lib\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.101450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101335    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-conf\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.101450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101374    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8rb\" (UniqueName: \"kubernetes.io/projected/9193a5a4-d86a-47f5-9db2-c23851803a36-kube-api-access-8n8rb\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:10:01.101846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101486    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.101846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101571    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.101846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101689    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.101846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101735    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhd95\" (UniqueName: \"kubernetes.io/projected/71f1519e-b03d-4ee5-9abd-b647e628531b-kube-api-access-fhd95\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.101846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101760    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.101846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101798    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.101846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101829    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42109964-a1d9-479b-b626-76b60f9f49da-tmp-dir\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:10:01.102192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101853    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6427a41a-4ada-4a89-a06a-b28b84815333-config-volume\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:01.102192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101876    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-config\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.102192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101898    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-systemd\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.102192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101923    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.102192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101931    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.102192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.101945    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9193a5a4-d86a-47f5-9db2-c23851803a36-serviceca\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:10:01.102192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102140    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.102192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102149    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102198    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lfx\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102224    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-systemd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102248    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102298    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6427a41a-4ada-4a89-a06a-b28b84815333-config-volume\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102343    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovn-node-metrics-cert\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102355    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovnkube-config\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102388    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102410    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42109964-a1d9-479b-b626-76b60f9f49da-tmp-dir\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102423    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-root\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102448    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-kubelet\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102491    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-secret-telemeter-client\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102505    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102532    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:10:01.102580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102558    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102592    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102619    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102756    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d2612a3-2331-4145-81bc-1c2eea9a37a5-cert\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102794    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-system-cni-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102841    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-federate-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102866    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-host\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102892    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-kubelet-config\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102921    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-ovn\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102948    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-env-overrides\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102974    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-sys-fs\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.102799    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103003    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103128    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9193a5a4-d86a-47f5-9db2-c23851803a36-serviceca\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:10:01.103224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103214    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b14437a8-5976-4417-99fe-481d0c2ea6b4-env-overrides\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103288    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103398    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414cbb04-b4d4-4840-911b-353f24025f02-metrics-certs\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103404    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103445    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b14437a8-5976-4417-99fe-481d0c2ea6b4-ovn-node-metrics-cert\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103455    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103492    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-federate-client-tls\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103500    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-daemon-config\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103560    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103574    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-kubernetes\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103680    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb66043-7c25-465f-95e4-3535cb4f35ff-host-slash\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103712    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5b4z\" (UniqueName: \"kubernetes.io/projected/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-kube-api-access-q5b4z\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103732    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-daemon-config\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103766    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-run\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103798    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103836    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-slash\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.103874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103862    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-log-socket\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.104680 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103890    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.104680 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.103918    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.104680 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.104132    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.104680 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.104228    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.104680 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.104322    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.104680 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.104475    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66b15cb4-0694-40c5-aeac-939d1635a93f-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:10:01.104953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.104903    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.105660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.105639    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.113941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.112537    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686tm\" (UniqueName: \"kubernetes.io/projected/6427a41a-4ada-4a89-a06a-b28b84815333-kube-api-access-686tm\") pod \"dns-default-zsrxr\" (UID: \"6427a41a-4ada-4a89-a06a-b28b84815333\") " pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:01.113941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.112606    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjfw\" (UniqueName: \"kubernetes.io/projected/f842fcd9-3de6-4162-a8f3-217a76df7e50-kube-api-access-mhjfw\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.113941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.112771    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqj4h\" (UniqueName: \"kubernetes.io/projected/4cb66043-7c25-465f-95e4-3535cb4f35ff-kube-api-access-lqj4h\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:10:01.113941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.112991    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8drfb\" (UniqueName: \"kubernetes.io/projected/ffea2206-5878-442e-a1fd-e1a691cfbcbc-kube-api-access-8drfb\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.113941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.113875    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9xhq\" (UniqueName: \"kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq\") pod \"console-786f578d97-qkqng\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") " pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:01.118082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.114347    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49dl\" (UniqueName: \"kubernetes.io/projected/414cbb04-b4d4-4840-911b-353f24025f02-kube-api-access-z49dl\") pod \"network-metrics-daemon-p67fc\" (UID: \"414cbb04-b4d4-4840-911b-353f24025f02\") " pod="openshift-multus/network-metrics-daemon-p67fc"
Oct 30 21:10:01.118082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.114865    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6drm\" (UniqueName: \"kubernetes.io/projected/9d2612a3-2331-4145-81bc-1c2eea9a37a5-kube-api-access-p6drm\") pod \"ingress-canary-krgh5\" (UID: \"9d2612a3-2331-4145-81bc-1c2eea9a37a5\") " pod="openshift-ingress-canary/ingress-canary-krgh5"
Oct 30 21:10:01.118082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.115623    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnk9\" (UniqueName: \"kubernetes.io/projected/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kube-api-access-tbnk9\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.118082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.117274    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgkv7\" (UniqueName: \"kubernetes.io/projected/66b15cb4-0694-40c5-aeac-939d1635a93f-kube-api-access-rgkv7\") pod \"prometheus-operator-9cd89b74c-kfdst\" (UID: \"66b15cb4-0694-40c5-aeac-939d1635a93f\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kfdst"
Oct 30 21:10:01.118082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.117343    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28qc\" (UniqueName: \"kubernetes.io/projected/4dbdcb7d-b158-484a-832b-30fa8cc62c53-kube-api-access-m28qc\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.118082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.117818    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjf8\" (UniqueName: \"kubernetes.io/projected/b14437a8-5976-4417-99fe-481d0c2ea6b4-kube-api-access-pmjf8\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.118371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.118149    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44gpq\" (UniqueName: \"kubernetes.io/projected/adf69dd8-b830-4e07-85d0-4950dcd2c9cf-kube-api-access-44gpq\") pod \"telemeter-client-59b7858fc8-fr72g\" (UID: \"adf69dd8-b830-4e07-85d0-4950dcd2c9cf\") " pod="openshift-monitoring/telemeter-client-59b7858fc8-fr72g"
Oct 30 21:10:01.118986 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.118960    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2bb\" (UniqueName: \"kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb\") pod \"console-b5fd76869-cqkmj\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") " pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:01.119062 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.118980    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmxj\" (UniqueName: \"kubernetes.io/projected/42109964-a1d9-479b-b626-76b60f9f49da-kube-api-access-8gmxj\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:10:01.121135 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.121113    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhd95\" (UniqueName: \"kubernetes.io/projected/71f1519e-b03d-4ee5-9abd-b647e628531b-kube-api-access-fhd95\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.121244 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.121117    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8rb\" (UniqueName: \"kubernetes.io/projected/9193a5a4-d86a-47f5-9db2-c23851803a36-kube-api-access-8n8rb\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:10:01.122262 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.122242    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8msz\" (UniqueName: \"kubernetes.io/projected/6f8e3f58-0a6e-41fa-a599-9d68ae65f01d-kube-api-access-n8msz\") pod \"network-check-target-vts2m\" (UID: \"6f8e3f58-0a6e-41fa-a599-9d68ae65f01d\") " pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:10:01.122395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.122378    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5b4z\" (UniqueName: \"kubernetes.io/projected/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-kube-api-access-q5b4z\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.122616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.122600    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lfx\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx\") pod \"prometheus-k8s-0\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:01.204543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204464    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-socket-dir-parent\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204502    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-lib-modules\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.204543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204520    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.204543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204542    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-wtmp\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204565    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-node-log\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204605    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-node-log\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204607    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-socket-dir-parent\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204616    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204629    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-node-exporter-wtmp\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204654    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-netns\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204663    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-lib-modules\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204681    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-system-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204706    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-netns\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204707    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-conf-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204732    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-multus\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204744    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-system-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204748    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-conf-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204747    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-registration-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204776    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-multus\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204786    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-registration-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204790    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-cnibin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204814    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysconfig\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204829    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-var-lib-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204830    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-cnibin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.204850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204844    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-var-lib-kubelet\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204873    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-bin\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204884    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysconfig\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204889    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cnibin\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204894    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-var-lib-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204910    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204926    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-bin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204929    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-bin\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204925    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-var-lib-kubelet\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204942    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-cnibin\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204944    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-socket-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204977    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-cni-bin\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204982    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-socket-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204984    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-kubelet\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.204986    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205016    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-modprobe-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205048    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-multus-certs\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205020    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-var-lib-kubelet\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205072    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42109964-a1d9-479b-b626-76b60f9f49da-hosts-file\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205084    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-multus-certs\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205099    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-dbus\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205111    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-modprobe-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205125    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205144    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-k8s-cni-cncf-io\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205171    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42109964-a1d9-479b-b626-76b60f9f49da-hosts-file\") pod \"node-resolver-tsz8h\" (UID: \"42109964-a1d9-479b-b626-76b60f9f49da\") " pod="openshift-dns/node-resolver-tsz8h"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205149    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-dbus\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205203    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-systemd-units\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205207    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205220    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-netd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205233    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-k8s-cni-cncf-io\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205240    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-systemd-units\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205261    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dbdcb7d-b158-484a-832b-30fa8cc62c53-crio-socket\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205278    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9193a5a4-d86a-47f5-9db2-c23851803a36-host\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205292    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-etc-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205277    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-cni-netd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205308    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-conf\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205328    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9193a5a4-d86a-47f5-9db2-c23851803a36-host\") pod \"node-ca-q5wbv\" (UID: \"9193a5a4-d86a-47f5-9db2-c23851803a36\") " pod="openshift-image-registry/node-ca-q5wbv"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205352    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-etc-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205355    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205374    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dbdcb7d-b158-484a-832b-30fa8cc62c53-crio-socket\") pod \"insights-runtime-extractor-gvjst\" (UID: \"4dbdcb7d-b158-484a-832b-30fa8cc62c53\") " pod="openshift-insights/insights-runtime-extractor-gvjst"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205393    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205418    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-conf\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205447    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-systemd\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205457    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-multus-cni-dir\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205454    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-sysctl-d\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205485    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-systemd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205484    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-systemd\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205519    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205525    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-systemd\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205538    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-root\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205565    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-root\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205585    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-openvswitch\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205612    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-kubelet\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205633    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205655    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-system-cni-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205657    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-kubelet\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205670    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205690    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffea2206-5878-442e-a1fd-e1a691cfbcbc-system-cni-dir\") pod \"multus-additional-cni-plugins-h2wqw\" (UID: \"ffea2206-5878-442e-a1fd-e1a691cfbcbc\") " pod="openshift-multus/multus-additional-cni-plugins-h2wqw"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205711    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-host\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205736    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-kubelet-config\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205757    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-ovn\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.205735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205771    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-host\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205796    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-sys-fs\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205803    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-run-ovn\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205816    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-kubernetes\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205825    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e7da34c-23be-4c24-977d-8ef4b007fe8f-kubelet-config\") pod \"global-pull-secret-syncer-kmwrj\" (UID: \"7e7da34c-23be-4c24-977d-8ef4b007fe8f\") " pod="kube-system/global-pull-secret-syncer-kmwrj"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205833    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb66043-7c25-465f-95e4-3535cb4f35ff-host-slash\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205842    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-sys-fs\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205849    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-run\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205870    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-etc-kubernetes\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205872    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205882    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-run\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205888    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cb66043-7c25-465f-95e4-3535cb4f35ff-host-slash\") pod \"iptables-alerter-slp8w\" (UID: \"4cb66043-7c25-465f-95e4-3535cb4f35ff\") " pod="openshift-network-operator/iptables-alerter-slp8w"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205895    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-slash\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205920    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205924    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-log-socket\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205932    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-host-slash\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205943    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14437a8-5976-4417-99fe-481d0c2ea6b4-log-socket\") pod \"ovnkube-node-tlfh6\" (UID: \"b14437a8-5976-4417-99fe-481d0c2ea6b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205955    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-netns\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205972    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-host-run-netns\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.205980    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-sys\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206025    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-hostroot\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206042    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-device-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206057    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-sys\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206081    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-etc-kubernetes\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206099    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-hostroot\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206132    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4e1ba50-dfdf-4627-8ab6-97d9b1598d51-sys\") pod \"node-exporter-7bnkf\" (UID: \"d4e1ba50-dfdf-4627-8ab6-97d9b1598d51\") " pod="openshift-monitoring/node-exporter-7bnkf"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206140    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d6d483a0-47e4-49a3-a59e-7e4fe09c8779-device-dir\") pod \"aws-ebs-csi-driver-node-rfjxw\" (UID: \"d6d483a0-47e4-49a3-a59e-7e4fe09c8779\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfjxw"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206171    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f842fcd9-3de6-4162-a8f3-217a76df7e50-sys\") pod \"tuned-99v8c\" (UID: \"f842fcd9-3de6-4162-a8f3-217a76df7e50\") " pod="openshift-cluster-node-tuning-operator/tuned-99v8c"
Oct 30 21:10:01.207606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.206196    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71f1519e-b03d-4ee5-9abd-b647e628531b-etc-kubernetes\") pod \"multus-w5hw9\" (UID: \"71f1519e-b03d-4ee5-9abd-b647e628531b\") " pod="openshift-multus/multus-w5hw9"
Oct 30 21:10:01.918334 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:01.918304    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:02.310546 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:02.310469    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zsrxr"
Oct 30 21:10:02.590214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:02.590112    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:02.609834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:02.609807    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:04.287601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:04.287574    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:04.288058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:04.287781    5886 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:10:04.843963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:04.843932    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:04.847721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:04.847694    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:08.848779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:08.848741    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:09.685245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:09.685208    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:10:11.026819 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:11.026736    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:10:15.967516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:15.967483    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n9vj8"
Oct 30 21:10:18.854303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:18.854277    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:27.058221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.058134    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-786f578d97-qkqng" podUID="c35b113e-843a-429a-883c-053ea13d9d0a" containerName="console" containerID="cri-o://ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278" gracePeriod=15
Oct 30 21:10:27.293349 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.293322    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:27.300129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300109    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp\") pod \"c35b113e-843a-429a-883c-053ea13d9d0a\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") "
Oct 30 21:10:27.300222 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300177    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert\") pod \"c35b113e-843a-429a-883c-053ea13d9d0a\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") "
Oct 30 21:10:27.300222 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300200    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca\") pod \"c35b113e-843a-429a-883c-053ea13d9d0a\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") "
Oct 30 21:10:27.300222 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300218    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config\") pod \"c35b113e-843a-429a-883c-053ea13d9d0a\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") "
Oct 30 21:10:27.300366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300238    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config\") pod \"c35b113e-843a-429a-883c-053ea13d9d0a\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") "
Oct 30 21:10:27.300366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300264    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert\") pod \"c35b113e-843a-429a-883c-053ea13d9d0a\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") "
Oct 30 21:10:27.300366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300303    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq\" (UniqueName: \"kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq\") pod \"c35b113e-843a-429a-883c-053ea13d9d0a\" (UID: \"c35b113e-843a-429a-883c-053ea13d9d0a\") "
Oct 30 21:10:27.300505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300376    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp" pod "c35b113e-843a-429a-883c-053ea13d9d0a" (UID: "c35b113e-843a-429a-883c-053ea13d9d0a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:27.300606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300587    5886 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35b113e-843a-429a-883c-053ea13d9d0a-tmp\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:27.300667 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300628    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca" pod "c35b113e-843a-429a-883c-053ea13d9d0a" (UID: "c35b113e-843a-429a-883c-053ea13d9d0a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:27.300727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300700    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert" pod "c35b113e-843a-429a-883c-053ea13d9d0a" (UID: "c35b113e-843a-429a-883c-053ea13d9d0a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:27.300794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.300733    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config" pod "c35b113e-843a-429a-883c-053ea13d9d0a" (UID: "c35b113e-843a-429a-883c-053ea13d9d0a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:27.302587 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.302559    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq" pod "c35b113e-843a-429a-883c-053ea13d9d0a" (UID: "c35b113e-843a-429a-883c-053ea13d9d0a"). InnerVolumeSpecName "kube-api-access-s9xhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:27.302713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.302588    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config" pod "c35b113e-843a-429a-883c-053ea13d9d0a" (UID: "c35b113e-843a-429a-883c-053ea13d9d0a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:27.302713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.302655    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert" pod "c35b113e-843a-429a-883c-053ea13d9d0a" (UID: "c35b113e-843a-429a-883c-053ea13d9d0a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:27.401166 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.401127    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9xhq\" (UniqueName: \"kubernetes.io/projected/c35b113e-843a-429a-883c-053ea13d9d0a-kube-api-access-s9xhq\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:27.401316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.401169    5886 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-oauth-serving-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:27.401316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.401182    5886 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:27.401316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.401191    5886 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-oauth-config\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:27.401316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.401200    5886 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35b113e-843a-429a-883c-053ea13d9d0a-console-config\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:27.401316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:27.401209    5886 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35b113e-843a-429a-883c-053ea13d9d0a-console-serving-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:28.128901 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.128866    5886 generic.go:356] "Generic (PLEG): container finished" podID="c35b113e-843a-429a-883c-053ea13d9d0a" containerID="ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278" exitCode=0
Oct 30 21:10:28.129333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.128909    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786f578d97-qkqng"
Oct 30 21:10:28.129333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.128947    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786f578d97-qkqng" event={"ID":"c35b113e-843a-429a-883c-053ea13d9d0a","Type":"ContainerDied","Data":"ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278"}
Oct 30 21:10:28.129333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.128995    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786f578d97-qkqng" event={"ID":"c35b113e-843a-429a-883c-053ea13d9d0a","Type":"ContainerDied","Data":"52be8e7c3a65736e0349b24059a842cee919e3da1a15b4535772212373c2bc5c"}
Oct 30 21:10:28.129333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.129015    5886 scope.go:122] "RemoveContainer" containerID="ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278"
Oct 30 21:10:28.137062 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.137042    5886 scope.go:122] "RemoveContainer" containerID="ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278"
Oct 30 21:10:28.137356 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:28.137337    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278\": container with ID starting with ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278 not found: ID does not exist" containerID="ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278"
Oct 30 21:10:28.137418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.137363    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278"} err="failed to get container status \"ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278\": rpc error: code = NotFound desc = could not find container \"ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278\": container with ID starting with ef9ae609974a5a32dd40b827b34776d24744187244e453fdd45d4ca7cb29f278 not found: ID does not exist"
Oct 30 21:10:28.177771 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.177745    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786f578d97-qkqng"]
Oct 30 21:10:28.189845 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:28.189818    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-786f578d97-qkqng"]
Oct 30 21:10:29.687950 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.687919    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vts2m"
Oct 30 21:10:29.963262 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.963177    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:10:29.963572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.963539    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="prometheus" containerID="cri-o://45bd60f4490cad10a4f87976d7566c85eda6eec05d66724c28708fa2427019f9" gracePeriod=600
Oct 30 21:10:29.963705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.963565    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy" containerID="cri-o://2910049e8b57a7c449466374993c815651f2b60866a41d5d25f42dca69bb05f8" gracePeriod=600
Oct 30 21:10:29.963705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.963590    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy-web" containerID="cri-o://7310404d7a12d4e9d0ca929f31096bae89d46797ebf1c5419986e221eaba86e9" gracePeriod=600
Oct 30 21:10:29.963705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.963624    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy-thanos" containerID="cri-o://967ba2f547ef89c5fbadfecec7e766669881d474c6ceaee98dd344ac994caa9f" gracePeriod=600
Oct 30 21:10:29.963705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.963584    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="thanos-sidecar" containerID="cri-o://76add9a95345cb5a0ce8a87ac880a7f6dc82385edc6dba95a38830082144ca05" gracePeriod=600
Oct 30 21:10:29.963705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.963605    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="config-reloader" containerID="cri-o://4d85d9becdf286edd7c10424da4af35274956952ec5b6ac258238c60db5a23b1" gracePeriod=600
Oct 30 21:10:29.987165 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:29.987125    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35b113e-843a-429a-883c-053ea13d9d0a" path="/var/lib/kubelet/pods/c35b113e-843a-429a-883c-053ea13d9d0a/volumes"
Oct 30 21:10:30.140743 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140707    5886 generic.go:356] "Generic (PLEG): container finished" podID="8bea9add-f465-49bf-ac0f-c2586e641917" containerID="967ba2f547ef89c5fbadfecec7e766669881d474c6ceaee98dd344ac994caa9f" exitCode=0
Oct 30 21:10:30.140743 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140740    5886 generic.go:356] "Generic (PLEG): container finished" podID="8bea9add-f465-49bf-ac0f-c2586e641917" containerID="2910049e8b57a7c449466374993c815651f2b60866a41d5d25f42dca69bb05f8" exitCode=0
Oct 30 21:10:30.140929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140754    5886 generic.go:356] "Generic (PLEG): container finished" podID="8bea9add-f465-49bf-ac0f-c2586e641917" containerID="7310404d7a12d4e9d0ca929f31096bae89d46797ebf1c5419986e221eaba86e9" exitCode=0
Oct 30 21:10:30.140929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140766    5886 generic.go:356] "Generic (PLEG): container finished" podID="8bea9add-f465-49bf-ac0f-c2586e641917" containerID="76add9a95345cb5a0ce8a87ac880a7f6dc82385edc6dba95a38830082144ca05" exitCode=0
Oct 30 21:10:30.140929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140778    5886 generic.go:356] "Generic (PLEG): container finished" podID="8bea9add-f465-49bf-ac0f-c2586e641917" containerID="4d85d9becdf286edd7c10424da4af35274956952ec5b6ac258238c60db5a23b1" exitCode=0
Oct 30 21:10:30.140929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140772    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerDied","Data":"967ba2f547ef89c5fbadfecec7e766669881d474c6ceaee98dd344ac994caa9f"}
Oct 30 21:10:30.140929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140790    5886 generic.go:356] "Generic (PLEG): container finished" podID="8bea9add-f465-49bf-ac0f-c2586e641917" containerID="45bd60f4490cad10a4f87976d7566c85eda6eec05d66724c28708fa2427019f9" exitCode=0
Oct 30 21:10:30.140929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140875    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerDied","Data":"2910049e8b57a7c449466374993c815651f2b60866a41d5d25f42dca69bb05f8"}
Oct 30 21:10:30.140929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140903    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerDied","Data":"7310404d7a12d4e9d0ca929f31096bae89d46797ebf1c5419986e221eaba86e9"}
Oct 30 21:10:30.140929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140927    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerDied","Data":"76add9a95345cb5a0ce8a87ac880a7f6dc82385edc6dba95a38830082144ca05"}
Oct 30 21:10:30.141178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140958    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerDied","Data":"4d85d9becdf286edd7c10424da4af35274956952ec5b6ac258238c60db5a23b1"}
Oct 30 21:10:30.141178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.140982    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerDied","Data":"45bd60f4490cad10a4f87976d7566c85eda6eec05d66724c28708fa2427019f9"}
Oct 30 21:10:30.240455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.240381    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:30.322361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322330    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322369    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322401    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322429    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322479    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322505    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322590    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322620    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322655    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322687    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322733    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322753    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322782    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322829    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322865    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322896    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322923    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.322969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322957    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.323642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.322990    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.323642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.323026    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config\") pod \"8bea9add-f465-49bf-ac0f-c2586e641917\" (UID: \"8bea9add-f465-49bf-ac0f-c2586e641917\") "
Oct 30 21:10:30.323642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.323106    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:30.323642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.323240    5886 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-metrics-client-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.324689 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.324648    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:30.325978 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.325945    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:30.326303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326276    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:30.326464 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326426    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:30.326540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326492    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:30.326540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326502    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.326540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326516    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:30.326723 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326661    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.326782 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326720    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:30.326841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326801    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.326915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.326887    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.327730 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.327704    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.328317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.328292    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:30.329054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.328998    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.329135 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.329083    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:30.329416 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.329392    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "kube-api-access-s4lfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:30.329892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.329872    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.330085 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.330065    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.336689 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.336668    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config" pod "8bea9add-f465-49bf-ac0f-c2586e641917" (UID: "8bea9add-f465-49bf-ac0f-c2586e641917"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:30.424510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424473    5886 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424508    5886 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-config\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424520    5886 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-tls-assets\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424542    5886 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424554    5886 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424567    5886 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424579    5886 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424592    5886 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424609    5886 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424621    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4lfx\" (UniqueName: \"kubernetes.io/projected/8bea9add-f465-49bf-ac0f-c2586e641917-kube-api-access-s4lfx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424634    5886 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-metrics-client-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424647    5886 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424659    5886 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-web-config\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424671    5886 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424683    5886 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-kube-rbac-proxy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424694    5886 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-config-out\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424706    5886 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424720    5886 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bea9add-f465-49bf-ac0f-c2586e641917-secret-grpc-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.424768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.424733    5886 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bea9add-f465-49bf-ac0f-c2586e641917-prometheus-k8s-db\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:30.458850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:30.453999    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlfh6"
Oct 30 21:10:31.147694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.147657    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bea9add-f465-49bf-ac0f-c2586e641917","Type":"ContainerDied","Data":"e9f44828ea5416868e085ae70cc40b9a944612b2654d11fd3b5a2d4337c75b80"}
Oct 30 21:10:31.147694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.147692    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.148234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.147761    5886 scope.go:122] "RemoveContainer" containerID="967ba2f547ef89c5fbadfecec7e766669881d474c6ceaee98dd344ac994caa9f"
Oct 30 21:10:31.158218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.158197    5886 scope.go:122] "RemoveContainer" containerID="2910049e8b57a7c449466374993c815651f2b60866a41d5d25f42dca69bb05f8"
Oct 30 21:10:31.164970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.164953    5886 scope.go:122] "RemoveContainer" containerID="7310404d7a12d4e9d0ca929f31096bae89d46797ebf1c5419986e221eaba86e9"
Oct 30 21:10:31.171227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.171211    5886 scope.go:122] "RemoveContainer" containerID="76add9a95345cb5a0ce8a87ac880a7f6dc82385edc6dba95a38830082144ca05"
Oct 30 21:10:31.177404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.177374    5886 scope.go:122] "RemoveContainer" containerID="4d85d9becdf286edd7c10424da4af35274956952ec5b6ac258238c60db5a23b1"
Oct 30 21:10:31.183700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.183684    5886 scope.go:122] "RemoveContainer" containerID="45bd60f4490cad10a4f87976d7566c85eda6eec05d66724c28708fa2427019f9"
Oct 30 21:10:31.190560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.190542    5886 scope.go:122] "RemoveContainer" containerID="e0702c68f1e9803de7ce4b24d262a10d472f1714a011375fadcce82c7bafa920"
Oct 30 21:10:31.210896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.210876    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:10:31.222276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.222258    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:10:31.240896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.240871    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:10:31.241113 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241100    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy-thanos"
Oct 30 21:10:31.241173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241115    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy-thanos"
Oct 30 21:10:31.241173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241128    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy"
Oct 30 21:10:31.241173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241133    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy"
Oct 30 21:10:31.241173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241140    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="prometheus"
Oct 30 21:10:31.241173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241145    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="prometheus"
Oct 30 21:10:31.241173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241171    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c35b113e-843a-429a-883c-053ea13d9d0a" containerName="console"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241177    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35b113e-843a-429a-883c-053ea13d9d0a" containerName="console"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241196    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy-web"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241201    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy-web"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241207    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="init-config-reloader"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241212    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="init-config-reloader"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241219    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="thanos-sidecar"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241224    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="thanos-sidecar"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241239    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="config-reloader"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241243    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="config-reloader"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241278    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="config-reloader"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241285    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy-thanos"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241291    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241296    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="prometheus"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241302    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35b113e-843a-429a-883c-053ea13d9d0a" containerName="console"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241307    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="kube-rbac-proxy-web"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241311    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="init-config-reloader"
Oct 30 21:10:31.241350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.241316    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" containerName="thanos-sidecar"
Oct 30 21:10:31.254579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.254561    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.274962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.274942    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:10:31.276022 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.276003    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:10:31.283140 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283101    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:10:31.283140 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283114    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:10:31.283503 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283483    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:10:31.283605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283542    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:10:31.283664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283605    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:10:31.283664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283640    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:10:31.283992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283949    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:10:31.283992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283968    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:10:31.283992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.283991    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:10:31.284189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.284011    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:10:31.284189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.284024    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:10:31.284189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.284032    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:10:31.284189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.284054    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4xf7\""
Oct 30 21:10:31.284189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.284068    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3q4vg4fcnnq04\""
Oct 30 21:10:31.299183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.297655    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:10:31.299727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.299710    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:10:31.330410 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330384    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330413    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e08318c-d1fa-4fcd-acd9-d9916c776598-config-out\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330433    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330449    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330621 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330564    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330625    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-config\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330681    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-web-config\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330704    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330725    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330790 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330750    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtfcp\" (UniqueName: \"kubernetes.io/projected/3e08318c-d1fa-4fcd-acd9-d9916c776598-kube-api-access-gtfcp\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330861 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330843    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330903    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330950 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330940    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e08318c-d1fa-4fcd-acd9-d9916c776598-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.330982 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.330966    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.331024 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.331011    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.331054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.331041    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.331103 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.331087    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.331139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.331131    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.331199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.331164    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.331199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.331191    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.407928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.407895    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5fd76869-cqkmj"]
Oct 30 21:10:31.432042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432011    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432057    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432080    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432105    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432128    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432180    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e08318c-d1fa-4fcd-acd9-d9916c776598-config-out\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432204    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432226    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432258    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432295    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-config\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432350    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-web-config\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432374    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432399    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432406    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432479    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtfcp\" (UniqueName: \"kubernetes.io/projected/3e08318c-d1fa-4fcd-acd9-d9916c776598-kube-api-access-gtfcp\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432516    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432522    5886 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432544    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.432584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432589    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e08318c-d1fa-4fcd-acd9-d9916c776598-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.433218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432618    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.433218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432649    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.433218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432669    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.433218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432683    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.433218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.432750    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.433487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.433449    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.434829 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.434196    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.434927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.434852    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.435501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.435475    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.436181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.436142    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.436614 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.436595    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3e08318c-d1fa-4fcd-acd9-d9916c776598-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.436867 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.436839    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.437064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.437046    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.437174 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.437137    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.437337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.437321    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e08318c-d1fa-4fcd-acd9-d9916c776598-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.438311 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.438285    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.438311 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.438309    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-config\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.438459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.438437    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.438519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.438453    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-web-config\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.439014 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.438991    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e08318c-d1fa-4fcd-acd9-d9916c776598-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.439388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.439368    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e08318c-d1fa-4fcd-acd9-d9916c776598-config-out\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.453976 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.453957    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtfcp\" (UniqueName: \"kubernetes.io/projected/3e08318c-d1fa-4fcd-acd9-d9916c776598-kube-api-access-gtfcp\") pod \"prometheus-k8s-0\" (UID: \"3e08318c-d1fa-4fcd-acd9-d9916c776598\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.658320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.658287    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:31.984521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:31.984448    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bea9add-f465-49bf-ac0f-c2586e641917" path="/var/lib/kubelet/pods/8bea9add-f465-49bf-ac0f-c2586e641917/volumes"
Oct 30 21:10:32.015082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:32.015054    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:10:32.021605 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:10:32.021568    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e08318c_d1fa_4fcd_acd9_d9916c776598.slice/crio-c8b32d134b2b62322b04be911700a1085d3312ae8784e5f5f5ca09065651ef0e WatchSource:0}: Error finding container c8b32d134b2b62322b04be911700a1085d3312ae8784e5f5f5ca09065651ef0e: Status 404 returned error can't find the container with id c8b32d134b2b62322b04be911700a1085d3312ae8784e5f5f5ca09065651ef0e
Oct 30 21:10:32.153116 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:32.153083    5886 generic.go:356] "Generic (PLEG): container finished" podID="3e08318c-d1fa-4fcd-acd9-d9916c776598" containerID="72775bc42493dcd18ea00ffa2560d0e3182aa81313a5cae5fac70f567bfe4746" exitCode=0
Oct 30 21:10:32.153452 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:32.153179    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e08318c-d1fa-4fcd-acd9-d9916c776598","Type":"ContainerDied","Data":"72775bc42493dcd18ea00ffa2560d0e3182aa81313a5cae5fac70f567bfe4746"}
Oct 30 21:10:32.153452 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:32.153248    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e08318c-d1fa-4fcd-acd9-d9916c776598","Type":"ContainerStarted","Data":"c8b32d134b2b62322b04be911700a1085d3312ae8784e5f5f5ca09065651ef0e"}
Oct 30 21:10:33.160076 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:33.160035    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e08318c-d1fa-4fcd-acd9-d9916c776598","Type":"ContainerStarted","Data":"978dd4849afec66b2e0e7f7602f25477188fe4e60cbfda73afadb6881065a68a"}
Oct 30 21:10:33.160555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:33.160123    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e08318c-d1fa-4fcd-acd9-d9916c776598","Type":"ContainerStarted","Data":"7df042a60bd5bf50f279fe989f4da195d92845caa6aabccef2b19f153c5b9409"}
Oct 30 21:10:33.160555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:33.160179    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e08318c-d1fa-4fcd-acd9-d9916c776598","Type":"ContainerStarted","Data":"348a85781298eca5e54b60782ad8c46d4e8991d0f8d6cd1388be07a7aaa3cfb9"}
Oct 30 21:10:33.160555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:33.160204    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e08318c-d1fa-4fcd-acd9-d9916c776598","Type":"ContainerStarted","Data":"19500a190241494cfecb7b4dc7315016d0ff2b67c2161cfc0b689bb2a2587c23"}
Oct 30 21:10:33.160555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:33.160227    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e08318c-d1fa-4fcd-acd9-d9916c776598","Type":"ContainerStarted","Data":"c3507fc174c9aac657876b502286c2d62ef3f0faacfda75b8f0cb06102b9778b"}
Oct 30 21:10:33.160555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:33.160250    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e08318c-d1fa-4fcd-acd9-d9916c776598","Type":"ContainerStarted","Data":"ca8117a866980af42f7ca877739235d53e2a43db38184b3463f506eb74879bbb"}
Oct 30 21:10:33.220517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:33.220459    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.205097919 podStartE2EDuration="2.220447376s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.349457ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:33.219419707 +0000 UTC m=+33.348671163" watchObservedRunningTime="2026-10-30 21:10:33.220447376 +0000 UTC m=+33.349698827"
Oct 30 21:10:36.659145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:36.659105    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:10:41.027609 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:41.027565    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:10:56.427074 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.427028    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-b5fd76869-cqkmj" podUID="8f6dad26-b908-4f23-9948-4dc9636e44f5" containerName="console" containerID="cri-o://9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4" gracePeriod=15
Oct 30 21:10:56.665027 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.665002    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:56.712955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.712881    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config\") pod \"8f6dad26-b908-4f23-9948-4dc9636e44f5\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") "
Oct 30 21:10:56.712955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.712920    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca\") pod \"8f6dad26-b908-4f23-9948-4dc9636e44f5\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") "
Oct 30 21:10:56.712955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.712947    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config\") pod \"8f6dad26-b908-4f23-9948-4dc9636e44f5\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") "
Oct 30 21:10:56.713245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.712983    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle\") pod \"8f6dad26-b908-4f23-9948-4dc9636e44f5\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") "
Oct 30 21:10:56.713245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713009    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp\") pod \"8f6dad26-b908-4f23-9948-4dc9636e44f5\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") "
Oct 30 21:10:56.713245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713035    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb\" (UniqueName: \"kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb\") pod \"8f6dad26-b908-4f23-9948-4dc9636e44f5\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") "
Oct 30 21:10:56.713245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713114    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert\") pod \"8f6dad26-b908-4f23-9948-4dc9636e44f5\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") "
Oct 30 21:10:56.713245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713140    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert\") pod \"8f6dad26-b908-4f23-9948-4dc9636e44f5\" (UID: \"8f6dad26-b908-4f23-9948-4dc9636e44f5\") "
Oct 30 21:10:56.713510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713347    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp" pod "8f6dad26-b908-4f23-9948-4dc9636e44f5" (UID: "8f6dad26-b908-4f23-9948-4dc9636e44f5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:10:56.713510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713370    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config" pod "8f6dad26-b908-4f23-9948-4dc9636e44f5" (UID: "8f6dad26-b908-4f23-9948-4dc9636e44f5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:56.713510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713375    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca" pod "8f6dad26-b908-4f23-9948-4dc9636e44f5" (UID: "8f6dad26-b908-4f23-9948-4dc9636e44f5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:56.713695 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713646    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert" pod "8f6dad26-b908-4f23-9948-4dc9636e44f5" (UID: "8f6dad26-b908-4f23-9948-4dc9636e44f5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:56.713767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.713741    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle" pod "8f6dad26-b908-4f23-9948-4dc9636e44f5" (UID: "8f6dad26-b908-4f23-9948-4dc9636e44f5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:10:56.715451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.715426    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert" pod "8f6dad26-b908-4f23-9948-4dc9636e44f5" (UID: "8f6dad26-b908-4f23-9948-4dc9636e44f5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:56.715804 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.715778    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config" pod "8f6dad26-b908-4f23-9948-4dc9636e44f5" (UID: "8f6dad26-b908-4f23-9948-4dc9636e44f5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:56.715894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.715811    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb" pod "8f6dad26-b908-4f23-9948-4dc9636e44f5" (UID: "8f6dad26-b908-4f23-9948-4dc9636e44f5"). InnerVolumeSpecName "kube-api-access-mt2bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:56.814433 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.814398    5886 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-oauth-serving-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:56.814433 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.814427    5886 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-serving-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:56.814433 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.814437    5886 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-config\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:56.814661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.814447    5886 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:56.814661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.814456    5886 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6dad26-b908-4f23-9948-4dc9636e44f5-console-oauth-config\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:56.814661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.814464    5886 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6dad26-b908-4f23-9948-4dc9636e44f5-trusted-ca-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:56.814661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.814473    5886 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f6dad26-b908-4f23-9948-4dc9636e44f5-tmp\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:56.814661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:56.814482    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt2bb\" (UniqueName: \"kubernetes.io/projected/8f6dad26-b908-4f23-9948-4dc9636e44f5-kube-api-access-mt2bb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:10:57.239476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.239442    5886 generic.go:356] "Generic (PLEG): container finished" podID="8f6dad26-b908-4f23-9948-4dc9636e44f5" containerID="9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4" exitCode=0
Oct 30 21:10:57.239676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.239483    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5fd76869-cqkmj"
Oct 30 21:10:57.239676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.239491    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5fd76869-cqkmj" event={"ID":"8f6dad26-b908-4f23-9948-4dc9636e44f5","Type":"ContainerDied","Data":"9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4"}
Oct 30 21:10:57.239676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.239540    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5fd76869-cqkmj" event={"ID":"8f6dad26-b908-4f23-9948-4dc9636e44f5","Type":"ContainerDied","Data":"73936e054d924a4fd3d155caad80a27fe1fb3ab05035278d14cc9de33a781485"}
Oct 30 21:10:57.239676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.239566    5886 scope.go:122] "RemoveContainer" containerID="9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4"
Oct 30 21:10:57.247595 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.247574    5886 scope.go:122] "RemoveContainer" containerID="9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4"
Oct 30 21:10:57.247843 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:10:57.247823    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4\": container with ID starting with 9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4 not found: ID does not exist" containerID="9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4"
Oct 30 21:10:57.247887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.247852    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4"} err="failed to get container status \"9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4\": rpc error: code = NotFound desc = could not find container \"9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4\": container with ID starting with 9d7ce8db10c077665bde63580feb834f6c399b20e3b1050973d1e5e11dc712e4 not found: ID does not exist"
Oct 30 21:10:57.288149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.288128    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5fd76869-cqkmj"]
Oct 30 21:10:57.299853 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.299831    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b5fd76869-cqkmj"]
Oct 30 21:10:57.984673 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:10:57.984639    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6dad26-b908-4f23-9948-4dc9636e44f5" path="/var/lib/kubelet/pods/8f6dad26-b908-4f23-9948-4dc9636e44f5/volumes"
Oct 30 21:11:11.027460 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:11:11.027418    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:11:14.327960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:11:14.327926    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:11:14.342299 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:11:14.342273    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:11:15.312132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:11:15.312104    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:11:41.026032 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:11:41.025940    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:12:11.026540 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:12:11.026500    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:12:41.027588 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:12:41.027548    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:13:11.025379 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:13:11.025289    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:13:41.026807 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:13:41.026767    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:14:11.027263 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:14:11.027220    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:14:41.027809 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:14:41.027724    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:14:47.851128 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.851094    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"]
Oct 30 21:14:47.851574 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.851378    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8f6dad26-b908-4f23-9948-4dc9636e44f5" containerName="console"
Oct 30 21:14:47.851574 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.851389    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6dad26-b908-4f23-9948-4dc9636e44f5" containerName="console"
Oct 30 21:14:47.851574 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.851431    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6dad26-b908-4f23-9948-4dc9636e44f5" containerName="console"
Oct 30 21:14:47.854404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.854383    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:47.870881 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.870853    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:14:47.873404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.873388    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:14:47.882401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.882383    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-mcbfl\""
Oct 30 21:14:47.885271 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.885250    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"]
Oct 30 21:14:47.951976 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.951944    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"]
Oct 30 21:14:47.954961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.954937    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pgrm\" (UniqueName: \"kubernetes.io/projected/ce3bb208-a679-4d46-9732-5a50b39d7e77-kube-api-access-6pgrm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:47.955081 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.954969    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:47.955081 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.954992    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:47.955151 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.955089    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:47.969694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.969671    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-mcxf8\""
Oct 30 21:14:47.974246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:47.974225    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"]
Oct 30 21:14:48.039966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.039938    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"]
Oct 30 21:14:48.044857 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.044825    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.056515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.056492    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jn4x\" (UniqueName: \"kubernetes.io/projected/a7df68ff-05de-47e7-9d34-93ea5325fac9-kube-api-access-7jn4x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.056860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.056833    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcjzz\" (UniqueName: \"kubernetes.io/projected/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-kube-api-access-zcjzz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.057047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057029    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.057273 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057243    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.057273 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057267    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.057418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057330    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.057478 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057423    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgrm\" (UniqueName: \"kubernetes.io/projected/ce3bb208-a679-4d46-9732-5a50b39d7e77-kube-api-access-6pgrm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:48.057614 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057592    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:48.057694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057645    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:48.057976 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057956    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:48.058026 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.057978    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:48.062282 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.062263    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-txnpc\""
Oct 30 21:14:48.062837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.062819    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"]
Oct 30 21:14:48.075683 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.075651    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgrm\" (UniqueName: \"kubernetes.io/projected/ce3bb208-a679-4d46-9732-5a50b39d7e77-kube-api-access-6pgrm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:48.135200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.135122    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"]
Oct 30 21:14:48.138305 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.138285    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.156924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.156902    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"]
Oct 30 21:14:48.158674 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.158653    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcjzz\" (UniqueName: \"kubernetes.io/projected/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-kube-api-access-zcjzz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.158780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.158703    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.158780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.158750    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.158780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.158772    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.158941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.158806    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.158941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.158865    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jn4x\" (UniqueName: \"kubernetes.io/projected/a7df68ff-05de-47e7-9d34-93ea5325fac9-kube-api-access-7jn4x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.158941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.158900    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.159062 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.158939    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.159109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.159074    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kghd\" (UniqueName: \"kubernetes.io/projected/519ddc33-c934-4e27-8476-3efe122c9630-kube-api-access-2kghd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.159213 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.159187    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.159368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.159198    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.159368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.159319    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.159483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.159433    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.162760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.162741    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-gwfb9\""
Oct 30 21:14:48.169878 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.169846    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:14:48.178397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.178373    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jn4x\" (UniqueName: \"kubernetes.io/projected/a7df68ff-05de-47e7-9d34-93ea5325fac9-kube-api-access-7jn4x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.178534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.178511    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcjzz\" (UniqueName: \"kubernetes.io/projected/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-kube-api-access-zcjzz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.259920 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.259886    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.260089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.259930    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.260089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.259950    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kghd\" (UniqueName: \"kubernetes.io/projected/519ddc33-c934-4e27-8476-3efe122c9630-kube-api-access-2kghd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.260304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.260281    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.260304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.260295    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.269932 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.269912    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:14:48.278246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.278211    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kghd\" (UniqueName: \"kubernetes.io/projected/519ddc33-c934-4e27-8476-3efe122c9630-kube-api-access-2kghd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.360608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.360571    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:14:48.453677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.453647    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:14:48.463962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.463939    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"]
Oct 30 21:14:48.469869 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:48.469844    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3bb208_a679_4d46_9732_5a50b39d7e77.slice/crio-d47111d0a36a877eb3ed0e3feb46dafe5f996bb3e2b669fbd75d7e4e87317325 WatchSource:0}: Error finding container d47111d0a36a877eb3ed0e3feb46dafe5f996bb3e2b669fbd75d7e4e87317325: Status 404 returned error can't find the container with id d47111d0a36a877eb3ed0e3feb46dafe5f996bb3e2b669fbd75d7e4e87317325
Oct 30 21:14:48.472076 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.472056    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:14:48.561908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.561877    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"]
Oct 30 21:14:48.567528 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:48.567504    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7df68ff_05de_47e7_9d34_93ea5325fac9.slice/crio-ed8246ca4b6ead6837d81efd031a9c5b81f23ca6011abf68fc494917d2b05710 WatchSource:0}: Error finding container ed8246ca4b6ead6837d81efd031a9c5b81f23ca6011abf68fc494917d2b05710: Status 404 returned error can't find the container with id ed8246ca4b6ead6837d81efd031a9c5b81f23ca6011abf68fc494917d2b05710
Oct 30 21:14:48.655647 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.655617    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"]
Oct 30 21:14:48.660949 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:48.660917    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad4f1c6_807b_482c_9bfc_d5ec98417d47.slice/crio-863115aa6c6f2c2a0f0481b25df229935ba90b05700b2ac1e28f7741bb8eec1a WatchSource:0}: Error finding container 863115aa6c6f2c2a0f0481b25df229935ba90b05700b2ac1e28f7741bb8eec1a: Status 404 returned error can't find the container with id 863115aa6c6f2c2a0f0481b25df229935ba90b05700b2ac1e28f7741bb8eec1a
Oct 30 21:14:48.727148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.727120    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"]
Oct 30 21:14:48.731671 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.731650    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.747335 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.747312    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"]
Oct 30 21:14:48.749486 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.749464    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-kxxhz\""
Oct 30 21:14:48.749802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.749784    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"]
Oct 30 21:14:48.754188 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:48.753313    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519ddc33_c934_4e27_8476_3efe122c9630.slice/crio-3f794c4d65ff19d1ffc3206f00181a58c8bd2e76cf5a0a7b334f415e23229808 WatchSource:0}: Error finding container 3f794c4d65ff19d1ffc3206f00181a58c8bd2e76cf5a0a7b334f415e23229808: Status 404 returned error can't find the container with id 3f794c4d65ff19d1ffc3206f00181a58c8bd2e76cf5a0a7b334f415e23229808
Oct 30 21:14:48.763349 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.763317    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkr7\" (UniqueName: \"kubernetes.io/projected/c479fdeb-725f-4db9-83d3-17deb68c90dd-kube-api-access-lhkr7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.763454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.763359    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.763454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.763383    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.864488 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.864446    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkr7\" (UniqueName: \"kubernetes.io/projected/c479fdeb-725f-4db9-83d3-17deb68c90dd-kube-api-access-lhkr7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.864819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.864503    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.864819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.864525    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.864892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.864830    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.864925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.864885    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.884132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.884108    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkr7\" (UniqueName: \"kubernetes.io/projected/c479fdeb-725f-4db9-83d3-17deb68c90dd-kube-api-access-lhkr7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:48.970302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.970224    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz" event={"ID":"2ad4f1c6-807b-482c-9bfc-d5ec98417d47","Type":"ContainerStarted","Data":"863115aa6c6f2c2a0f0481b25df229935ba90b05700b2ac1e28f7741bb8eec1a"}
Oct 30 21:14:48.971690 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.971659    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv" event={"ID":"a7df68ff-05de-47e7-9d34-93ea5325fac9","Type":"ContainerStarted","Data":"ed8246ca4b6ead6837d81efd031a9c5b81f23ca6011abf68fc494917d2b05710"}
Oct 30 21:14:48.972984 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.972950    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm" event={"ID":"ce3bb208-a679-4d46-9732-5a50b39d7e77","Type":"ContainerStarted","Data":"d47111d0a36a877eb3ed0e3feb46dafe5f996bb3e2b669fbd75d7e4e87317325"}
Oct 30 21:14:48.974233 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:48.974207    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq" event={"ID":"519ddc33-c934-4e27-8476-3efe122c9630","Type":"ContainerStarted","Data":"3f794c4d65ff19d1ffc3206f00181a58c8bd2e76cf5a0a7b334f415e23229808"}
Oct 30 21:14:49.047835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.047798    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:14:49.363232 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.363131    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"]
Oct 30 21:14:49.372104 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:49.371487    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc479fdeb_725f_4db9_83d3_17deb68c90dd.slice/crio-2b0e36c5cc094512b31ebaf09dc48fff96060be4ea8b11b9347d1cc770e51881 WatchSource:0}: Error finding container 2b0e36c5cc094512b31ebaf09dc48fff96060be4ea8b11b9347d1cc770e51881: Status 404 returned error can't find the container with id 2b0e36c5cc094512b31ebaf09dc48fff96060be4ea8b11b9347d1cc770e51881
Oct 30 21:14:49.810883 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.810838    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"]
Oct 30 21:14:49.815930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.815903    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.845104 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.845072    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"]
Oct 30 21:14:49.845312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.845293    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gsbcc\""
Oct 30 21:14:49.873994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.873911    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4dc7\" (UniqueName: \"kubernetes.io/projected/f01401cf-1caa-429c-b098-80b7175ae8dc-kube-api-access-m4dc7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.873994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.873958    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.874548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.874011    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.974468 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.974427    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4dc7\" (UniqueName: \"kubernetes.io/projected/f01401cf-1caa-429c-b098-80b7175ae8dc-kube-api-access-m4dc7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.974655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.974502    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.974655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.974589    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.975480 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.975454    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.975593 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.975569    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:49.993687 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.993620    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns" event={"ID":"c479fdeb-725f-4db9-83d3-17deb68c90dd","Type":"ContainerStarted","Data":"2b0e36c5cc094512b31ebaf09dc48fff96060be4ea8b11b9347d1cc770e51881"}
Oct 30 21:14:49.993687 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:49.993637    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4dc7\" (UniqueName: \"kubernetes.io/projected/f01401cf-1caa-429c-b098-80b7175ae8dc-kube-api-access-m4dc7\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:50.137489 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:50.137413    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:14:50.483008 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:50.482975    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"]
Oct 30 21:14:50.528266 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:50.490939    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01401cf_1caa_429c_b098_80b7175ae8dc.slice/crio-a817a7e5c2bfcb14b514d1ed5cbec042a2064bdb23cb0a67066f4190060e444c WatchSource:0}: Error finding container a817a7e5c2bfcb14b514d1ed5cbec042a2064bdb23cb0a67066f4190060e444c: Status 404 returned error can't find the container with id a817a7e5c2bfcb14b514d1ed5cbec042a2064bdb23cb0a67066f4190060e444c
Oct 30 21:14:50.999521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:50.998950    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7" event={"ID":"f01401cf-1caa-429c-b098-80b7175ae8dc","Type":"ContainerStarted","Data":"a817a7e5c2bfcb14b514d1ed5cbec042a2064bdb23cb0a67066f4190060e444c"}
Oct 30 21:14:52.829292 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:52.829257    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"]
Oct 30 21:14:52.834294 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:52.833674    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:52.858851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:52.858796    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-q44lp\""
Oct 30 21:14:52.860113 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:52.860083    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"]
Oct 30 21:14:52.899257 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:52.899219    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:52.899432 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:52.899316    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hl72\" (UniqueName: \"kubernetes.io/projected/3d3b0638-af95-483d-80c2-c9dabd561cbc-kube-api-access-4hl72\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:52.899432 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:52.899353    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:53.000276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.000236    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hl72\" (UniqueName: \"kubernetes.io/projected/3d3b0638-af95-483d-80c2-c9dabd561cbc-kube-api-access-4hl72\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:53.000449 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.000290    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:53.000449 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.000339    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:53.000934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.000879    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:53.001616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.001585    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:53.033230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.033198    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hl72\" (UniqueName: \"kubernetes.io/projected/3d3b0638-af95-483d-80c2-c9dabd561cbc-kube-api-access-4hl72\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:53.156540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.156501    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:14:53.478903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.478807    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"]
Oct 30 21:14:53.487917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.487887    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.488787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.488758    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"]
Oct 30 21:14:53.503383 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:53.503357    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3b0638_af95_483d_80c2_c9dabd561cbc.slice/crio-1ebcfa0978260ec38c523605478e1a748810274fb26a5445765017fca75e6cef WatchSource:0}: Error finding container 1ebcfa0978260ec38c523605478e1a748810274fb26a5445765017fca75e6cef: Status 404 returned error can't find the container with id 1ebcfa0978260ec38c523605478e1a748810274fb26a5445765017fca75e6cef
Oct 30 21:14:53.504766 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.504744    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:14:53.505110 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505087    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.505203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505128    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.505203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505188    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg27r\" (UniqueName: \"kubernetes.io/projected/f9457827-beb1-4755-bb94-29ac9c2c1659-kube-api-access-cg27r\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.505303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505226    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.505303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505259    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.505303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505283    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.505465 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505317    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.505465 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505351    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.505465 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.505383    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9457827-beb1-4755-bb94-29ac9c2c1659-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.508078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.508055    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:14:53.508183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.508131    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:14:53.513293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.513250    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c2rf5\""
Oct 30 21:14:53.514331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.514254    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"]
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.605929    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.605983    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.606018    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9457827-beb1-4755-bb94-29ac9c2c1659-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.606085    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.606113    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.606151    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg27r\" (UniqueName: \"kubernetes.io/projected/f9457827-beb1-4755-bb94-29ac9c2c1659-kube-api-access-cg27r\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.606203    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.606239    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.607180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.606264    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.610921 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.608686    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.610921 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.609627    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.610921 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.610109    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.610921 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.610210    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.610921 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.610679    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9457827-beb1-4755-bb94-29ac9c2c1659-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.614737 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.614693    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.622543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.622518    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.638548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.638517    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg27r\" (UniqueName: \"kubernetes.io/projected/f9457827-beb1-4755-bb94-29ac9c2c1659-kube-api-access-cg27r\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.639020 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.638977    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9457827-beb1-4755-bb94-29ac9c2c1659-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cf7lb\" (UID: \"f9457827-beb1-4755-bb94-29ac9c2c1659\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:53.820150 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:53.820071    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:14:54.029484 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.029382    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb" event={"ID":"3d3b0638-af95-483d-80c2-c9dabd561cbc","Type":"ContainerStarted","Data":"1ebcfa0978260ec38c523605478e1a748810274fb26a5445765017fca75e6cef"}
Oct 30 21:14:54.202515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.202473    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"]
Oct 30 21:14:54.208221 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:54.208186    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9457827_beb1_4755_bb94_29ac9c2c1659.slice/crio-60d17c5bec1066399e98603c5f6ca797a3405f745237a7364e94ca936ab4f8e0 WatchSource:0}: Error finding container 60d17c5bec1066399e98603c5f6ca797a3405f745237a7364e94ca936ab4f8e0: Status 404 returned error can't find the container with id 60d17c5bec1066399e98603c5f6ca797a3405f745237a7364e94ca936ab4f8e0
Oct 30 21:14:54.808604 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.808561    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"]
Oct 30 21:14:54.813970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.813458    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.827048 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.827023    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.827216 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.827081    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.827216 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.827197    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7tx\" (UniqueName: \"kubernetes.io/projected/5fd7bbde-feba-4082-b02f-ddf374836f9d-kube-api-access-rq7tx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.842293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.842261    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"]
Oct 30 21:14:54.842439 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.842321    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"]
Oct 30 21:14:54.848767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.848402    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:54.865765 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.865701    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"]
Oct 30 21:14:54.929610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.928271    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvsxh\" (UniqueName: \"kubernetes.io/projected/0a2e6687-1ec6-479d-93aa-ce126b18c391-kube-api-access-dvsxh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:54.929610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.928349    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7tx\" (UniqueName: \"kubernetes.io/projected/5fd7bbde-feba-4082-b02f-ddf374836f9d-kube-api-access-rq7tx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.929610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.928387    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:54.929610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.928419    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.929610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.928461    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:54.929610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.928491    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.929610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.928884    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.929610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.929513    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:54.947418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:54.947394    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7tx\" (UniqueName: \"kubernetes.io/projected/5fd7bbde-feba-4082-b02f-ddf374836f9d-kube-api-access-rq7tx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:55.029886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.029854    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:55.030359 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.029903    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:55.030359 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.029956    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvsxh\" (UniqueName: \"kubernetes.io/projected/0a2e6687-1ec6-479d-93aa-ce126b18c391-kube-api-access-dvsxh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:55.030542 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.030521    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:55.030731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.030707    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:55.070492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.070419    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvsxh\" (UniqueName: \"kubernetes.io/projected/0a2e6687-1ec6-479d-93aa-ce126b18c391-kube-api-access-dvsxh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:55.074544 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.074504    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" event={"ID":"f9457827-beb1-4755-bb94-29ac9c2c1659","Type":"ContainerStarted","Data":"60d17c5bec1066399e98603c5f6ca797a3405f745237a7364e94ca936ab4f8e0"}
Oct 30 21:14:55.138716 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.138672    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:14:55.170634 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.170587    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:14:55.510379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.510297    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"]
Oct 30 21:14:55.530346 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:55.529126    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd7bbde_feba_4082_b02f_ddf374836f9d.slice/crio-def3f951e132b6efc5237d386690ba27b982365f040e8be328b3d9dcf9481850 WatchSource:0}: Error finding container def3f951e132b6efc5237d386690ba27b982365f040e8be328b3d9dcf9481850: Status 404 returned error can't find the container with id def3f951e132b6efc5237d386690ba27b982365f040e8be328b3d9dcf9481850
Oct 30 21:14:55.530615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:55.530492    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"]
Oct 30 21:14:55.536400 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:14:55.536370    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2e6687_1ec6_479d_93aa_ce126b18c391.slice/crio-5e52023ed834a83f86a4287f8ec081bb5943c1931672139a81a9855a72358ae3 WatchSource:0}: Error finding container 5e52023ed834a83f86a4287f8ec081bb5943c1931672139a81a9855a72358ae3: Status 404 returned error can't find the container with id 5e52023ed834a83f86a4287f8ec081bb5943c1931672139a81a9855a72358ae3
Oct 30 21:14:56.095245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:56.095203    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958" event={"ID":"5fd7bbde-feba-4082-b02f-ddf374836f9d","Type":"ContainerStarted","Data":"def3f951e132b6efc5237d386690ba27b982365f040e8be328b3d9dcf9481850"}
Oct 30 21:14:56.098364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:14:56.098304    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p" event={"ID":"0a2e6687-1ec6-479d-93aa-ce126b18c391","Type":"ContainerStarted","Data":"5e52023ed834a83f86a4287f8ec081bb5943c1931672139a81a9855a72358ae3"}
Oct 30 21:15:03.222718 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:03.222670    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:03.223188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:03.222770    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:03.223188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:03.222815    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:15:04.201640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:04.200672    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" event={"ID":"f9457827-beb1-4755-bb94-29ac9c2c1659","Type":"ContainerStarted","Data":"5cef17eb5485a66ac428b90fa63008b4b093994a6351446613bad310a99a40bd"}
Oct 30 21:15:04.204530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:04.204497    5886 generic.go:356] "Generic (PLEG): container finished" podID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerID="35f346150b0c899d3c788e82e22c69b95931fe13c141940a9d644d92c62b430a" exitCode=0
Oct 30 21:15:04.204667 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:04.204605    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm" event={"ID":"ce3bb208-a679-4d46-9732-5a50b39d7e77","Type":"ContainerDied","Data":"35f346150b0c899d3c788e82e22c69b95931fe13c141940a9d644d92c62b430a"}
Oct 30 21:15:04.262386 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:04.262318    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" podStartSLOduration=2.2537823599999998 podStartE2EDuration="11.262300376s" totalImagesPullingTime="9.008518016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:04.251183889 +0000 UTC m=+304.380435343" watchObservedRunningTime="2026-10-30 21:15:04.262300376 +0000 UTC m=+304.391551836"
Oct 30 21:15:04.828186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:04.820954    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:15:04.828186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:04.827440    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:15:05.213224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:05.213181    5886 generic.go:356] "Generic (PLEG): container finished" podID="519ddc33-c934-4e27-8476-3efe122c9630" containerID="896db00ba9b507a66fac435456dc938a57df68db79bb2adc3df419a7b9112435" exitCode=0
Oct 30 21:15:05.213364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:05.213346    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq" event={"ID":"519ddc33-c934-4e27-8476-3efe122c9630","Type":"ContainerDied","Data":"896db00ba9b507a66fac435456dc938a57df68db79bb2adc3df419a7b9112435"}
Oct 30 21:15:05.216785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:05.216758    5886 generic.go:356] "Generic (PLEG): container finished" podID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerID="be9e092ab145052ff0ff8e9a5b22f74e6c849fe971bf3cda2a515538d3946061" exitCode=0
Oct 30 21:15:05.216884 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:05.216844    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv" event={"ID":"a7df68ff-05de-47e7-9d34-93ea5325fac9","Type":"ContainerDied","Data":"be9e092ab145052ff0ff8e9a5b22f74e6c849fe971bf3cda2a515538d3946061"}
Oct 30 21:15:05.219719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:05.219692    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz" event={"ID":"2ad4f1c6-807b-482c-9bfc-d5ec98417d47","Type":"ContainerStarted","Data":"823856cfb7043390f7cd23e84f6309554560a66f2fc811a99c5475209da7a74d"}
Oct 30 21:15:05.220629 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:05.220610    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:15:05.221913 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:05.221840    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:15:06.225252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:06.225146    5886 generic.go:356] "Generic (PLEG): container finished" podID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerID="2ab5b9fecf6f5c5f4cb16e520964e8873fb75266af417df280038c446e1a27ec" exitCode=0
Oct 30 21:15:06.225252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:06.225194    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb" event={"ID":"3d3b0638-af95-483d-80c2-c9dabd561cbc","Type":"ContainerDied","Data":"2ab5b9fecf6f5c5f4cb16e520964e8873fb75266af417df280038c446e1a27ec"}
Oct 30 21:15:06.226892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:06.226852    5886 generic.go:356] "Generic (PLEG): container finished" podID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerID="823856cfb7043390f7cd23e84f6309554560a66f2fc811a99c5475209da7a74d" exitCode=0
Oct 30 21:15:06.227006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:06.226960    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz" event={"ID":"2ad4f1c6-807b-482c-9bfc-d5ec98417d47","Type":"ContainerDied","Data":"823856cfb7043390f7cd23e84f6309554560a66f2fc811a99c5475209da7a74d"}
Oct 30 21:15:06.228649 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:06.228551    5886 generic.go:356] "Generic (PLEG): container finished" podID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerID="877dc9a6219712d5e2908ecbcc9b4f8d5e86d6a0333fda9862b8b39e69057950" exitCode=0
Oct 30 21:15:06.228749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:06.228652    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958" event={"ID":"5fd7bbde-feba-4082-b02f-ddf374836f9d","Type":"ContainerDied","Data":"877dc9a6219712d5e2908ecbcc9b4f8d5e86d6a0333fda9862b8b39e69057950"}
Oct 30 21:15:07.244948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.240557    5886 generic.go:356] "Generic (PLEG): container finished" podID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerID="2425fc0b4dc683c5c5d8da5a6759a1ffa1f7a73fc2203c176701fa0800c829b4" exitCode=0
Oct 30 21:15:07.244948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.240676    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm" event={"ID":"ce3bb208-a679-4d46-9732-5a50b39d7e77","Type":"ContainerDied","Data":"2425fc0b4dc683c5c5d8da5a6759a1ffa1f7a73fc2203c176701fa0800c829b4"}
Oct 30 21:15:07.247374 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.247344    5886 generic.go:356] "Generic (PLEG): container finished" podID="519ddc33-c934-4e27-8476-3efe122c9630" containerID="b6f6189231d5cbed163986c613d413385c4516fafce21c659cbae75929fa6610" exitCode=0
Oct 30 21:15:07.247467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.247434    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq" event={"ID":"519ddc33-c934-4e27-8476-3efe122c9630","Type":"ContainerDied","Data":"b6f6189231d5cbed163986c613d413385c4516fafce21c659cbae75929fa6610"}
Oct 30 21:15:07.249856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.249829    5886 generic.go:356] "Generic (PLEG): container finished" podID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerID="c6d59970a5c13061b244fa1e55f67d26de190a58eada4378493911b6b586247a" exitCode=0
Oct 30 21:15:07.249965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.249941    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7" event={"ID":"f01401cf-1caa-429c-b098-80b7175ae8dc","Type":"ContainerDied","Data":"c6d59970a5c13061b244fa1e55f67d26de190a58eada4378493911b6b586247a"}
Oct 30 21:15:07.251703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.251683    5886 generic.go:356] "Generic (PLEG): container finished" podID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerID="a8b8973ebd3650377c0cad727f5618188de6cfc73462e2d145a8c4f20c1613c6" exitCode=0
Oct 30 21:15:07.251806 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.251758    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns" event={"ID":"c479fdeb-725f-4db9-83d3-17deb68c90dd","Type":"ContainerDied","Data":"a8b8973ebd3650377c0cad727f5618188de6cfc73462e2d145a8c4f20c1613c6"}
Oct 30 21:15:07.265269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.262242    5886 generic.go:356] "Generic (PLEG): container finished" podID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerID="a5635551c7b5ec993de9a63b99cb631124e431b39e551858ab7a1ec28b62f61e" exitCode=0
Oct 30 21:15:07.265269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.262327    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p" event={"ID":"0a2e6687-1ec6-479d-93aa-ce126b18c391","Type":"ContainerDied","Data":"a5635551c7b5ec993de9a63b99cb631124e431b39e551858ab7a1ec28b62f61e"}
Oct 30 21:15:07.266635 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.265931    5886 generic.go:356] "Generic (PLEG): container finished" podID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerID="c85145b164439391bd984a3f2f8e3960445f10e9fefcb98cd23da559f61e72ba" exitCode=0
Oct 30 21:15:07.266635 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:07.266060    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958" event={"ID":"5fd7bbde-feba-4082-b02f-ddf374836f9d","Type":"ContainerDied","Data":"c85145b164439391bd984a3f2f8e3960445f10e9fefcb98cd23da559f61e72ba"}
Oct 30 21:15:08.274102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:08.274059    5886 generic.go:356] "Generic (PLEG): container finished" podID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerID="56d4d55c03841a958f2712cff51820dc68dadf6514772868e706a9cfe0c8e410" exitCode=0
Oct 30 21:15:08.274596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:08.274198    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns" event={"ID":"c479fdeb-725f-4db9-83d3-17deb68c90dd","Type":"ContainerDied","Data":"56d4d55c03841a958f2712cff51820dc68dadf6514772868e706a9cfe0c8e410"}
Oct 30 21:15:08.279499 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:08.279470    5886 generic.go:356] "Generic (PLEG): container finished" podID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerID="e2a9631d73a58ec01611b24321d365ca0a3b65eeec8fb85f547e8f89a60b4145" exitCode=0
Oct 30 21:15:08.279596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:08.279556    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv" event={"ID":"a7df68ff-05de-47e7-9d34-93ea5325fac9","Type":"ContainerDied","Data":"e2a9631d73a58ec01611b24321d365ca0a3b65eeec8fb85f547e8f89a60b4145"}
Oct 30 21:15:08.283829 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:08.283796    5886 generic.go:356] "Generic (PLEG): container finished" podID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerID="90901342308a0403e1eb1a31599698001b43969ec6e176f318a50e871b00c9b4" exitCode=0
Oct 30 21:15:08.283933 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:08.283883    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz" event={"ID":"2ad4f1c6-807b-482c-9bfc-d5ec98417d47","Type":"ContainerDied","Data":"90901342308a0403e1eb1a31599698001b43969ec6e176f318a50e871b00c9b4"}
Oct 30 21:15:08.288386 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:08.287705    5886 generic.go:356] "Generic (PLEG): container finished" podID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerID="686108d0cf53b09d9df9573936d53d7cddcd2a79f1f6add851945b39aabf2b93" exitCode=0
Oct 30 21:15:08.288386 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:08.287773    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p" event={"ID":"0a2e6687-1ec6-479d-93aa-ce126b18c391","Type":"ContainerDied","Data":"686108d0cf53b09d9df9573936d53d7cddcd2a79f1f6add851945b39aabf2b93"}
Oct 30 21:15:10.311803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:10.311756    5886 generic.go:356] "Generic (PLEG): container finished" podID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerID="196bfd20f2e5d10bf7ab8f26a5c8390962af87d587578c175c3887d13b53b523" exitCode=0
Oct 30 21:15:10.312272 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:10.311818    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb" event={"ID":"3d3b0638-af95-483d-80c2-c9dabd561cbc","Type":"ContainerDied","Data":"196bfd20f2e5d10bf7ab8f26a5c8390962af87d587578c175c3887d13b53b523"}
Oct 30 21:15:11.063690 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:15:11.063617    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:15:18.426645 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:18.425648    5886 generic.go:356] "Generic (PLEG): container finished" podID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerID="8acc969dfa4797e5111d96fccd6f786e9056415f6b5dbe3f0bfba34aec4db91d" exitCode=0
Oct 30 21:15:18.426645 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:18.425764    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7" event={"ID":"f01401cf-1caa-429c-b098-80b7175ae8dc","Type":"ContainerDied","Data":"8acc969dfa4797e5111d96fccd6f786e9056415f6b5dbe3f0bfba34aec4db91d"}
Oct 30 21:15:24.490636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:24.490549    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958" event={"ID":"5fd7bbde-feba-4082-b02f-ddf374836f9d","Type":"ContainerStarted","Data":"470eb566ccfc23cad0ad18c4b47fb57bda5f2697a6f7c47c01638a0bfaec79e7"}
Oct 30 21:15:24.495533 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:24.495505    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm" event={"ID":"ce3bb208-a679-4d46-9732-5a50b39d7e77","Type":"ContainerStarted","Data":"6bcdd03feb2a30cf3dde3150c6bfbe138ccabe3084ab5041a0cd5b7e9d2650c1"}
Oct 30 21:15:24.531455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:24.531389    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958" podStartSLOduration=2.461647446 podStartE2EDuration="30.531368951s" totalImagesPullingTime="28.069721505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:24.5302011 +0000 UTC m=+324.659452561" watchObservedRunningTime="2026-10-30 21:15:24.531368951 +0000 UTC m=+324.660620409"
Oct 30 21:15:24.569401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:24.569351    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm" podStartSLOduration=3.362583565 podStartE2EDuration="37.569336373s" totalImagesPullingTime="34.206752808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:24.568332338 +0000 UTC m=+324.697583789" watchObservedRunningTime="2026-10-30 21:15:24.569336373 +0000 UTC m=+324.698587826"
Oct 30 21:15:25.520132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:25.520091    5886 generic.go:356] "Generic (PLEG): container finished" podID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerID="470eb566ccfc23cad0ad18c4b47fb57bda5f2697a6f7c47c01638a0bfaec79e7" exitCode=0
Oct 30 21:15:25.520598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:25.520197    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958" event={"ID":"5fd7bbde-feba-4082-b02f-ddf374836f9d","Type":"ContainerDied","Data":"470eb566ccfc23cad0ad18c4b47fb57bda5f2697a6f7c47c01638a0bfaec79e7"}
Oct 30 21:15:25.526398 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:25.526363    5886 generic.go:356] "Generic (PLEG): container finished" podID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerID="6bcdd03feb2a30cf3dde3150c6bfbe138ccabe3084ab5041a0cd5b7e9d2650c1" exitCode=0
Oct 30 21:15:25.526547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:25.526476    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm" event={"ID":"ce3bb208-a679-4d46-9732-5a50b39d7e77","Type":"ContainerDied","Data":"6bcdd03feb2a30cf3dde3150c6bfbe138ccabe3084ab5041a0cd5b7e9d2650c1"}
Oct 30 21:15:25.530533 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:25.530328    5886 generic.go:356] "Generic (PLEG): container finished" podID="519ddc33-c934-4e27-8476-3efe122c9630" containerID="dd6999ceedf8647cda26fd3957c1bb62342ae6fc917d02f33939e5b932e13d75" exitCode=0
Oct 30 21:15:25.530533 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:25.530389    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq" event={"ID":"519ddc33-c934-4e27-8476-3efe122c9630","Type":"ContainerDied","Data":"dd6999ceedf8647cda26fd3957c1bb62342ae6fc917d02f33939e5b932e13d75"}
Oct 30 21:15:26.538087 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.538055    5886 generic.go:356] "Generic (PLEG): container finished" podID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerID="f705a8bdc92ddd11cab2dd935de58f9ef96cfc5499c4da5da74ae28e49711698" exitCode=0
Oct 30 21:15:26.538498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.538181    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv" event={"ID":"a7df68ff-05de-47e7-9d34-93ea5325fac9","Type":"ContainerDied","Data":"f705a8bdc92ddd11cab2dd935de58f9ef96cfc5499c4da5da74ae28e49711698"}
Oct 30 21:15:26.541175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.541128    5886 generic.go:356] "Generic (PLEG): container finished" podID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerID="06abcae15d1e0e355234cc3b0b82bb0c1001ba461232cfc1d0d6ee6fb81bed2b" exitCode=0
Oct 30 21:15:26.541295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.541197    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz" event={"ID":"2ad4f1c6-807b-482c-9bfc-d5ec98417d47","Type":"ContainerDied","Data":"06abcae15d1e0e355234cc3b0b82bb0c1001ba461232cfc1d0d6ee6fb81bed2b"}
Oct 30 21:15:26.763912 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.763884    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:15:26.786264 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.786239    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:15:26.804605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.804577    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:15:26.869227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.869191    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-bundle\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-bundle\") pod \"519ddc33-c934-4e27-8476-3efe122c9630\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") "
Oct 30 21:15:26.869390 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.869282    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-util\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-util\") pod \"519ddc33-c934-4e27-8476-3efe122c9630\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") "
Oct 30 21:15:26.869390 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.869317    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519ddc33-c934-4e27-8476-3efe122c9630-kube-api-access-2kghd\" (UniqueName: \"kubernetes.io/projected/519ddc33-c934-4e27-8476-3efe122c9630-kube-api-access-2kghd\") pod \"519ddc33-c934-4e27-8476-3efe122c9630\" (UID: \"519ddc33-c934-4e27-8476-3efe122c9630\") "
Oct 30 21:15:26.869938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.869908    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-bundle" pod "519ddc33-c934-4e27-8476-3efe122c9630" (UID: "519ddc33-c934-4e27-8476-3efe122c9630"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:26.872902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.872872    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519ddc33-c934-4e27-8476-3efe122c9630-kube-api-access-2kghd" pod "519ddc33-c934-4e27-8476-3efe122c9630" (UID: "519ddc33-c934-4e27-8476-3efe122c9630"). InnerVolumeSpecName "kube-api-access-2kghd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:26.877922 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.877893    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-util" pod "519ddc33-c934-4e27-8476-3efe122c9630" (UID: "519ddc33-c934-4e27-8476-3efe122c9630"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:26.970752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.970673    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd7bbde-feba-4082-b02f-ddf374836f9d-kube-api-access-rq7tx\" (UniqueName: \"kubernetes.io/projected/5fd7bbde-feba-4082-b02f-ddf374836f9d-kube-api-access-rq7tx\") pod \"5fd7bbde-feba-4082-b02f-ddf374836f9d\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") "
Oct 30 21:15:26.970752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.970730    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3bb208-a679-4d46-9732-5a50b39d7e77-kube-api-access-6pgrm\" (UniqueName: \"kubernetes.io/projected/ce3bb208-a679-4d46-9732-5a50b39d7e77-kube-api-access-6pgrm\") pod \"ce3bb208-a679-4d46-9732-5a50b39d7e77\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") "
Oct 30 21:15:26.970954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.970773    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-util\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-util\") pod \"ce3bb208-a679-4d46-9732-5a50b39d7e77\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") "
Oct 30 21:15:26.970954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.970803    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-bundle\") pod \"5fd7bbde-feba-4082-b02f-ddf374836f9d\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") "
Oct 30 21:15:26.970954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.970877    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-bundle\") pod \"ce3bb208-a679-4d46-9732-5a50b39d7e77\" (UID: \"ce3bb208-a679-4d46-9732-5a50b39d7e77\") "
Oct 30 21:15:26.970954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.970925    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-util\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-util\") pod \"5fd7bbde-feba-4082-b02f-ddf374836f9d\" (UID: \"5fd7bbde-feba-4082-b02f-ddf374836f9d\") "
Oct 30 21:15:26.971190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.971149    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kghd\" (UniqueName: \"kubernetes.io/projected/519ddc33-c934-4e27-8476-3efe122c9630-kube-api-access-2kghd\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:26.971239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.971196    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:26.971239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.971211    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/519ddc33-c934-4e27-8476-3efe122c9630-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:26.972530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.972482    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-bundle" pod "ce3bb208-a679-4d46-9732-5a50b39d7e77" (UID: "ce3bb208-a679-4d46-9732-5a50b39d7e77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:26.972659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.972604    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-bundle" pod "5fd7bbde-feba-4082-b02f-ddf374836f9d" (UID: "5fd7bbde-feba-4082-b02f-ddf374836f9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:26.974926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.974894    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd7bbde-feba-4082-b02f-ddf374836f9d-kube-api-access-rq7tx" pod "5fd7bbde-feba-4082-b02f-ddf374836f9d" (UID: "5fd7bbde-feba-4082-b02f-ddf374836f9d"). InnerVolumeSpecName "kube-api-access-rq7tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:26.975321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.975293    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3bb208-a679-4d46-9732-5a50b39d7e77-kube-api-access-6pgrm" pod "ce3bb208-a679-4d46-9732-5a50b39d7e77" (UID: "ce3bb208-a679-4d46-9732-5a50b39d7e77"). InnerVolumeSpecName "kube-api-access-6pgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:26.977260 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.977233    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-util" pod "ce3bb208-a679-4d46-9732-5a50b39d7e77" (UID: "ce3bb208-a679-4d46-9732-5a50b39d7e77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:26.979717 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:26.979691    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-util" pod "5fd7bbde-feba-4082-b02f-ddf374836f9d" (UID: "5fd7bbde-feba-4082-b02f-ddf374836f9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:27.072575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.072536    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq7tx\" (UniqueName: \"kubernetes.io/projected/5fd7bbde-feba-4082-b02f-ddf374836f9d-kube-api-access-rq7tx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.072575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.072576    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pgrm\" (UniqueName: \"kubernetes.io/projected/ce3bb208-a679-4d46-9732-5a50b39d7e77-kube-api-access-6pgrm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.072802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.072588    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.072802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.072605    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.072802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.072616    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3bb208-a679-4d46-9732-5a50b39d7e77-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.072802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.072624    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fd7bbde-feba-4082-b02f-ddf374836f9d-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.547058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.547021    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq"
Oct 30 21:15:27.547058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.547030    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8mgfq" event={"ID":"519ddc33-c934-4e27-8476-3efe122c9630","Type":"ContainerDied","Data":"3f794c4d65ff19d1ffc3206f00181a58c8bd2e76cf5a0a7b334f415e23229808"}
Oct 30 21:15:27.547451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.547075    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f794c4d65ff19d1ffc3206f00181a58c8bd2e76cf5a0a7b334f415e23229808"
Oct 30 21:15:27.549578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.549549    5886 generic.go:356] "Generic (PLEG): container finished" podID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerID="4cc3d6c168199c909a75d3077f3b4a6cf7a831debd6b8dbe0466869cad7b40be" exitCode=0
Oct 30 21:15:27.549683 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.549643    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns" event={"ID":"c479fdeb-725f-4db9-83d3-17deb68c90dd","Type":"ContainerDied","Data":"4cc3d6c168199c909a75d3077f3b4a6cf7a831debd6b8dbe0466869cad7b40be"}
Oct 30 21:15:27.551816 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.551796    5886 generic.go:356] "Generic (PLEG): container finished" podID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerID="a9535f0edaa83fc1f1ad3d7d148cd92b98c70ac4bf64fb8de5e175ab1f7de575" exitCode=0
Oct 30 21:15:27.551917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.551872    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p" event={"ID":"0a2e6687-1ec6-479d-93aa-ce126b18c391","Type":"ContainerDied","Data":"a9535f0edaa83fc1f1ad3d7d148cd92b98c70ac4bf64fb8de5e175ab1f7de575"}
Oct 30 21:15:27.553969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.553951    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958" event={"ID":"5fd7bbde-feba-4082-b02f-ddf374836f9d","Type":"ContainerDied","Data":"def3f951e132b6efc5237d386690ba27b982365f040e8be328b3d9dcf9481850"}
Oct 30 21:15:27.554061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.553975    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acf958"
Oct 30 21:15:27.554061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.553981    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="def3f951e132b6efc5237d386690ba27b982365f040e8be328b3d9dcf9481850"
Oct 30 21:15:27.555921 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.555899    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm" event={"ID":"ce3bb208-a679-4d46-9732-5a50b39d7e77","Type":"ContainerDied","Data":"d47111d0a36a877eb3ed0e3feb46dafe5f996bb3e2b669fbd75d7e4e87317325"}
Oct 30 21:15:27.555998 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.555939    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d47111d0a36a877eb3ed0e3feb46dafe5f996bb3e2b669fbd75d7e4e87317325"
Oct 30 21:15:27.555998 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.555944    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vpkxm"
Oct 30 21:15:27.702358 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.702337    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:15:27.736848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.736817    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:15:27.879688 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.879657    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-util\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-util\") pod \"a7df68ff-05de-47e7-9d34-93ea5325fac9\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") "
Oct 30 21:15:27.879799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.879698    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7df68ff-05de-47e7-9d34-93ea5325fac9-kube-api-access-7jn4x\" (UniqueName: \"kubernetes.io/projected/a7df68ff-05de-47e7-9d34-93ea5325fac9-kube-api-access-7jn4x\") pod \"a7df68ff-05de-47e7-9d34-93ea5325fac9\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") "
Oct 30 21:15:27.879799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.879728    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-util\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-util\") pod \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") "
Oct 30 21:15:27.879799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.879785    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-bundle\") pod \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") "
Oct 30 21:15:27.879952 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.879842    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-bundle\") pod \"a7df68ff-05de-47e7-9d34-93ea5325fac9\" (UID: \"a7df68ff-05de-47e7-9d34-93ea5325fac9\") "
Oct 30 21:15:27.879952 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.879878    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-kube-api-access-zcjzz\" (UniqueName: \"kubernetes.io/projected/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-kube-api-access-zcjzz\") pod \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\" (UID: \"2ad4f1c6-807b-482c-9bfc-d5ec98417d47\") "
Oct 30 21:15:27.880328 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.880294    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-bundle" pod "2ad4f1c6-807b-482c-9bfc-d5ec98417d47" (UID: "2ad4f1c6-807b-482c-9bfc-d5ec98417d47"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:27.880443 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.880411    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-bundle" pod "a7df68ff-05de-47e7-9d34-93ea5325fac9" (UID: "a7df68ff-05de-47e7-9d34-93ea5325fac9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:27.881870 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.881846    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-kube-api-access-zcjzz" pod "2ad4f1c6-807b-482c-9bfc-d5ec98417d47" (UID: "2ad4f1c6-807b-482c-9bfc-d5ec98417d47"). InnerVolumeSpecName "kube-api-access-zcjzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:27.882058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.882037    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7df68ff-05de-47e7-9d34-93ea5325fac9-kube-api-access-7jn4x" pod "a7df68ff-05de-47e7-9d34-93ea5325fac9" (UID: "a7df68ff-05de-47e7-9d34-93ea5325fac9"). InnerVolumeSpecName "kube-api-access-7jn4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:27.885610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.885587    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-util" pod "2ad4f1c6-807b-482c-9bfc-d5ec98417d47" (UID: "2ad4f1c6-807b-482c-9bfc-d5ec98417d47"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:27.885972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.885953    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-util" pod "a7df68ff-05de-47e7-9d34-93ea5325fac9" (UID: "a7df68ff-05de-47e7-9d34-93ea5325fac9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:27.981437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.981365    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.981437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.981398    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcjzz\" (UniqueName: \"kubernetes.io/projected/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-kube-api-access-zcjzz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.981437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.981412    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7df68ff-05de-47e7-9d34-93ea5325fac9-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.981437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.981425    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jn4x\" (UniqueName: \"kubernetes.io/projected/a7df68ff-05de-47e7-9d34-93ea5325fac9-kube-api-access-7jn4x\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.981437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.981437    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:27.981678 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:27.981451    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ad4f1c6-807b-482c-9bfc-d5ec98417d47-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:28.561436 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.561403    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv" event={"ID":"a7df68ff-05de-47e7-9d34-93ea5325fac9","Type":"ContainerDied","Data":"ed8246ca4b6ead6837d81efd031a9c5b81f23ca6011abf68fc494917d2b05710"}
Oct 30 21:15:28.561841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.561461    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed8246ca4b6ead6837d81efd031a9c5b81f23ca6011abf68fc494917d2b05710"
Oct 30 21:15:28.561841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.561415    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwxsv"
Oct 30 21:15:28.563808 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.563775    5886 generic.go:356] "Generic (PLEG): container finished" podID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerID="a97440dc2e2700eb31516f06b79ecef03a4fb966f06b54a4e686070725af1b04" exitCode=0
Oct 30 21:15:28.563922 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.563859    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb" event={"ID":"3d3b0638-af95-483d-80c2-c9dabd561cbc","Type":"ContainerDied","Data":"a97440dc2e2700eb31516f06b79ecef03a4fb966f06b54a4e686070725af1b04"}
Oct 30 21:15:28.565571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.565553    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz"
Oct 30 21:15:28.565670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.565554    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64kjcz" event={"ID":"2ad4f1c6-807b-482c-9bfc-d5ec98417d47","Type":"ContainerDied","Data":"863115aa6c6f2c2a0f0481b25df229935ba90b05700b2ac1e28f7741bb8eec1a"}
Oct 30 21:15:28.565710 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.565666    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="863115aa6c6f2c2a0f0481b25df229935ba90b05700b2ac1e28f7741bb8eec1a"
Oct 30 21:15:28.567696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.567676    5886 generic.go:356] "Generic (PLEG): container finished" podID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerID="8ee72a157576ab22107c85b954876ec4888885d5e17df9ca80714a10dc90e467" exitCode=0
Oct 30 21:15:28.567790 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.567720    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7" event={"ID":"f01401cf-1caa-429c-b098-80b7175ae8dc","Type":"ContainerDied","Data":"8ee72a157576ab22107c85b954876ec4888885d5e17df9ca80714a10dc90e467"}
Oct 30 21:15:28.695206 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.695128    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:15:28.721656 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.721634    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:15:28.888417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.888380    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c479fdeb-725f-4db9-83d3-17deb68c90dd-kube-api-access-lhkr7\" (UniqueName: \"kubernetes.io/projected/c479fdeb-725f-4db9-83d3-17deb68c90dd-kube-api-access-lhkr7\") pod \"c479fdeb-725f-4db9-83d3-17deb68c90dd\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") "
Oct 30 21:15:28.888417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.888426    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-util\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-util\") pod \"0a2e6687-1ec6-479d-93aa-ce126b18c391\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") "
Oct 30 21:15:28.888670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.888460    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-util\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-util\") pod \"c479fdeb-725f-4db9-83d3-17deb68c90dd\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") "
Oct 30 21:15:28.888670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.888482    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-bundle\") pod \"c479fdeb-725f-4db9-83d3-17deb68c90dd\" (UID: \"c479fdeb-725f-4db9-83d3-17deb68c90dd\") "
Oct 30 21:15:28.888670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.888499    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-bundle\") pod \"0a2e6687-1ec6-479d-93aa-ce126b18c391\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") "
Oct 30 21:15:28.888670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.888575    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2e6687-1ec6-479d-93aa-ce126b18c391-kube-api-access-dvsxh\" (UniqueName: \"kubernetes.io/projected/0a2e6687-1ec6-479d-93aa-ce126b18c391-kube-api-access-dvsxh\") pod \"0a2e6687-1ec6-479d-93aa-ce126b18c391\" (UID: \"0a2e6687-1ec6-479d-93aa-ce126b18c391\") "
Oct 30 21:15:28.889121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.889085    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-bundle" pod "0a2e6687-1ec6-479d-93aa-ce126b18c391" (UID: "0a2e6687-1ec6-479d-93aa-ce126b18c391"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:28.889280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.889183    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-bundle" pod "c479fdeb-725f-4db9-83d3-17deb68c90dd" (UID: "c479fdeb-725f-4db9-83d3-17deb68c90dd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:28.890601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.890582    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c479fdeb-725f-4db9-83d3-17deb68c90dd-kube-api-access-lhkr7" pod "c479fdeb-725f-4db9-83d3-17deb68c90dd" (UID: "c479fdeb-725f-4db9-83d3-17deb68c90dd"). InnerVolumeSpecName "kube-api-access-lhkr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:28.891033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.891011    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2e6687-1ec6-479d-93aa-ce126b18c391-kube-api-access-dvsxh" pod "0a2e6687-1ec6-479d-93aa-ce126b18c391" (UID: "0a2e6687-1ec6-479d-93aa-ce126b18c391"). InnerVolumeSpecName "kube-api-access-dvsxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:28.894798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.894773    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-util" pod "c479fdeb-725f-4db9-83d3-17deb68c90dd" (UID: "c479fdeb-725f-4db9-83d3-17deb68c90dd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:28.895137 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.895117    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-util" pod "0a2e6687-1ec6-479d-93aa-ce126b18c391" (UID: "0a2e6687-1ec6-479d-93aa-ce126b18c391"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:28.989956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.989866    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvsxh\" (UniqueName: \"kubernetes.io/projected/0a2e6687-1ec6-479d-93aa-ce126b18c391-kube-api-access-dvsxh\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:28.989956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.989896    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkr7\" (UniqueName: \"kubernetes.io/projected/c479fdeb-725f-4db9-83d3-17deb68c90dd-kube-api-access-lhkr7\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:28.989956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.989908    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:28.989956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.989919    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:28.989956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.989930    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c479fdeb-725f-4db9-83d3-17deb68c90dd-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:28.989956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:28.989940    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a2e6687-1ec6-479d-93aa-ce126b18c391-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:29.573082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.573051    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns"
Oct 30 21:15:29.573082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.573068    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tj7ns" event={"ID":"c479fdeb-725f-4db9-83d3-17deb68c90dd","Type":"ContainerDied","Data":"2b0e36c5cc094512b31ebaf09dc48fff96060be4ea8b11b9347d1cc770e51881"}
Oct 30 21:15:29.573685 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.573121    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b0e36c5cc094512b31ebaf09dc48fff96060be4ea8b11b9347d1cc770e51881"
Oct 30 21:15:29.575033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.575014    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p"
Oct 30 21:15:29.575145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.575056    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xx79p" event={"ID":"0a2e6687-1ec6-479d-93aa-ce126b18c391","Type":"ContainerDied","Data":"5e52023ed834a83f86a4287f8ec081bb5943c1931672139a81a9855a72358ae3"}
Oct 30 21:15:29.575145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.575099    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e52023ed834a83f86a4287f8ec081bb5943c1931672139a81a9855a72358ae3"
Oct 30 21:15:29.705501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.705440    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:15:29.730514 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.730485    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:15:29.897265 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.897228    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-bundle\") pod \"f01401cf-1caa-429c-b098-80b7175ae8dc\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") "
Oct 30 21:15:29.897265 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.897266    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-util\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-util\") pod \"3d3b0638-af95-483d-80c2-c9dabd561cbc\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") "
Oct 30 21:15:29.897488 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.897292    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-util\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-util\") pod \"f01401cf-1caa-429c-b098-80b7175ae8dc\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") "
Oct 30 21:15:29.897488 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.897332    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01401cf-1caa-429c-b098-80b7175ae8dc-kube-api-access-m4dc7\" (UniqueName: \"kubernetes.io/projected/f01401cf-1caa-429c-b098-80b7175ae8dc-kube-api-access-m4dc7\") pod \"f01401cf-1caa-429c-b098-80b7175ae8dc\" (UID: \"f01401cf-1caa-429c-b098-80b7175ae8dc\") "
Oct 30 21:15:29.897488 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.897411    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3b0638-af95-483d-80c2-c9dabd561cbc-kube-api-access-4hl72\" (UniqueName: \"kubernetes.io/projected/3d3b0638-af95-483d-80c2-c9dabd561cbc-kube-api-access-4hl72\") pod \"3d3b0638-af95-483d-80c2-c9dabd561cbc\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") "
Oct 30 21:15:29.897488 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.897454    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-bundle\") pod \"3d3b0638-af95-483d-80c2-c9dabd561cbc\" (UID: \"3d3b0638-af95-483d-80c2-c9dabd561cbc\") "
Oct 30 21:15:29.897759 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.897708    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-bundle" pod "f01401cf-1caa-429c-b098-80b7175ae8dc" (UID: "f01401cf-1caa-429c-b098-80b7175ae8dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:29.898221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.898179    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-bundle" pod "3d3b0638-af95-483d-80c2-c9dabd561cbc" (UID: "3d3b0638-af95-483d-80c2-c9dabd561cbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:29.899586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.899560    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3b0638-af95-483d-80c2-c9dabd561cbc-kube-api-access-4hl72" pod "3d3b0638-af95-483d-80c2-c9dabd561cbc" (UID: "3d3b0638-af95-483d-80c2-c9dabd561cbc"). InnerVolumeSpecName "kube-api-access-4hl72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:29.899689 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.899675    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01401cf-1caa-429c-b098-80b7175ae8dc-kube-api-access-m4dc7" pod "f01401cf-1caa-429c-b098-80b7175ae8dc" (UID: "f01401cf-1caa-429c-b098-80b7175ae8dc"). InnerVolumeSpecName "kube-api-access-m4dc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:29.905288 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.905261    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-util" pod "f01401cf-1caa-429c-b098-80b7175ae8dc" (UID: "f01401cf-1caa-429c-b098-80b7175ae8dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:29.905353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.905337    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-util" pod "3d3b0638-af95-483d-80c2-c9dabd561cbc" (UID: "3d3b0638-af95-483d-80c2-c9dabd561cbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:29.998567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.998543    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hl72\" (UniqueName: \"kubernetes.io/projected/3d3b0638-af95-483d-80c2-c9dabd561cbc-kube-api-access-4hl72\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:29.998567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.998567    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:29.998696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.998577    5886 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:29.998696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.998584    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d3b0638-af95-483d-80c2-c9dabd561cbc-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:29.998696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.998592    5886 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f01401cf-1caa-429c-b098-80b7175ae8dc-util\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:29.998696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:29.998599    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4dc7\" (UniqueName: \"kubernetes.io/projected/f01401cf-1caa-429c-b098-80b7175ae8dc-kube-api-access-m4dc7\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:15:30.580886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:30.580857    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb"
Oct 30 21:15:30.581383 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:30.580893    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpdcjb" event={"ID":"3d3b0638-af95-483d-80c2-c9dabd561cbc","Type":"ContainerDied","Data":"1ebcfa0978260ec38c523605478e1a748810274fb26a5445765017fca75e6cef"}
Oct 30 21:15:30.581383 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:30.580941    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ebcfa0978260ec38c523605478e1a748810274fb26a5445765017fca75e6cef"
Oct 30 21:15:30.582930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:30.582899    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7" event={"ID":"f01401cf-1caa-429c-b098-80b7175ae8dc","Type":"ContainerDied","Data":"a817a7e5c2bfcb14b514d1ed5cbec042a2064bdb23cb0a67066f4190060e444c"}
Oct 30 21:15:30.583092 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:30.582934    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tw4b7"
Oct 30 21:15:30.583092 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:30.582936    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a817a7e5c2bfcb14b514d1ed5cbec042a2064bdb23cb0a67066f4190060e444c"
Oct 30 21:15:32.538683 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538653    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"]
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538916    5886 cpu_manager.go:414] "containerMap: removing container" podUID="519ddc33-c934-4e27-8476-3efe122c9630" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538927    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519ddc33-c934-4e27-8476-3efe122c9630" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538936    5886 cpu_manager.go:414] "containerMap: removing container" podUID="519ddc33-c934-4e27-8476-3efe122c9630" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538941    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519ddc33-c934-4e27-8476-3efe122c9630" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538951    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538956    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538962    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538968    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538973    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538978    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538984    5886 cpu_manager.go:414] "containerMap: removing container" podUID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerName="extract"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538988    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerName="extract"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.538996    5886 cpu_manager.go:414] "containerMap: removing container" podUID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539000    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539006    5886 cpu_manager.go:414] "containerMap: removing container" podUID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerName="extract"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539010    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerName="extract"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539017    5886 cpu_manager.go:414] "containerMap: removing container" podUID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539022    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539030    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539034    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539045    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539049    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539055    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539058    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539066    5886 cpu_manager.go:414] "containerMap: removing container" podUID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539071    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerName="util"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539077    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerName="extract"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539081    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerName="extract"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539087    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerName="pull"
Oct 30 21:15:32.539079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539091    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerName="pull"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539098    5886 cpu_manager.go:414] "containerMap: removing container" podUID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerName="pull"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539103    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerName="pull"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539109    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539113    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539118    5886 cpu_manager.go:414] "containerMap: removing container" podUID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539122    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539129    5886 cpu_manager.go:414] "containerMap: removing container" podUID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerName="util"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539134    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerName="util"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539142    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerName="util"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539146    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerName="util"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539163    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerName="util"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539168    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerName="util"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539174    5886 cpu_manager.go:414] "containerMap: removing container" podUID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerName="pull"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539178    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerName="pull"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539184    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerName="pull"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539188    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerName="pull"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539194    5886 cpu_manager.go:414] "containerMap: removing container" podUID="519ddc33-c934-4e27-8476-3efe122c9630" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539198    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519ddc33-c934-4e27-8476-3efe122c9630" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539204    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539208    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539213    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539217    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539224    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539228    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539267    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2e6687-1ec6-479d-93aa-ce126b18c391" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539274    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd7bbde-feba-4082-b02f-ddf374836f9d" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539282    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7df68ff-05de-47e7-9d34-93ea5325fac9" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539288    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3b0638-af95-483d-80c2-c9dabd561cbc" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539292    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3bb208-a679-4d46-9732-5a50b39d7e77" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539298    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01401cf-1caa-429c-b098-80b7175ae8dc" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539303    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="519ddc33-c934-4e27-8476-3efe122c9630" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539307    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="c479fdeb-725f-4db9-83d3-17deb68c90dd" containerName="extract"
Oct 30 21:15:32.539951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.539313    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad4f1c6-807b-482c-9bfc-d5ec98417d47" containerName="extract"
Oct 30 21:15:32.543560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.543536    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.562799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.562771    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:15:32.562799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.562668    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:15:32.562995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.562814    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:15:32.563264 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.563237    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:15:32.563393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.563271    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:15:32.573761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.573742    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dggpd\""
Oct 30 21:15:32.579775 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.579751    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"]
Oct 30 21:15:32.721002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.720970    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.721002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.721002    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/115d18d1-9c71-4b73-a9ab-00101d5f96cf-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.721236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.721094    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22rqx\" (UniqueName: \"kubernetes.io/projected/115d18d1-9c71-4b73-a9ab-00101d5f96cf-kube-api-access-22rqx\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.721236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.721133    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/115d18d1-9c71-4b73-a9ab-00101d5f96cf-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.721236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.721196    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.721333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.721259    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.821971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.821881    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22rqx\" (UniqueName: \"kubernetes.io/projected/115d18d1-9c71-4b73-a9ab-00101d5f96cf-kube-api-access-22rqx\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.821971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.821916    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/115d18d1-9c71-4b73-a9ab-00101d5f96cf-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.821971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.821932    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.822203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.822031    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.822203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.822062    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.822203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.822077    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/115d18d1-9c71-4b73-a9ab-00101d5f96cf-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.822590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.822551    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/115d18d1-9c71-4b73-a9ab-00101d5f96cf-manifests\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.824588 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.824561    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/115d18d1-9c71-4b73-a9ab-00101d5f96cf-tmp\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.824717 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.824599    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.824787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.824766    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.824844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.824773    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/115d18d1-9c71-4b73-a9ab-00101d5f96cf-metrics-cert\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.841615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.841590    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22rqx\" (UniqueName: \"kubernetes.io/projected/115d18d1-9c71-4b73-a9ab-00101d5f96cf-kube-api-access-22rqx\") pod \"opendatahub-operator-controller-manager-864fdc7bd9-bpjf7\" (UID: \"115d18d1-9c71-4b73-a9ab-00101d5f96cf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:32.858485 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:32.858463    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:33.219790 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:33.219755    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"]
Oct 30 21:15:33.224407 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:15:33.224376    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115d18d1_9c71_4b73_a9ab_00101d5f96cf.slice/crio-7434aa428058ea91d64a59e9127374d2b356cc8b52b0339f3de37ed20c6237fc WatchSource:0}: Error finding container 7434aa428058ea91d64a59e9127374d2b356cc8b52b0339f3de37ed20c6237fc: Status 404 returned error can't find the container with id 7434aa428058ea91d64a59e9127374d2b356cc8b52b0339f3de37ed20c6237fc
Oct 30 21:15:33.595937 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:33.595854    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7" event={"ID":"115d18d1-9c71-4b73-a9ab-00101d5f96cf","Type":"ContainerStarted","Data":"7434aa428058ea91d64a59e9127374d2b356cc8b52b0339f3de37ed20c6237fc"}
Oct 30 21:15:36.612673 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:36.612638    5886 generic.go:356] "Generic (PLEG): container finished" podID="115d18d1-9c71-4b73-a9ab-00101d5f96cf" containerID="c37650ce864c2f302964fbf63a9823f63649e07e3aee9b0833d59600b53a0667" exitCode=0
Oct 30 21:15:36.613073 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:36.612714    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7" event={"ID":"115d18d1-9c71-4b73-a9ab-00101d5f96cf","Type":"ContainerDied","Data":"c37650ce864c2f302964fbf63a9823f63649e07e3aee9b0833d59600b53a0667"}
Oct 30 21:15:37.619551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:37.619511    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7" event={"ID":"115d18d1-9c71-4b73-a9ab-00101d5f96cf","Type":"ContainerStarted","Data":"b719ec86206fec20df6d034baae24c88b1da6d4021a96be2b5ccacfef282ea48"}
Oct 30 21:15:37.619974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:37.619619    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:37.701604 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:37.701555    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7" podStartSLOduration=2.070254384 podStartE2EDuration="5.701538859s" totalImagesPullingTime="3.057305457s" totalInitContainerRuntime="573.979018ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:37.689096426 +0000 UTC m=+337.818347885" watchObservedRunningTime="2026-10-30 21:15:37.701538859 +0000 UTC m=+337.830790308"
Oct 30 21:15:41.027683 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:15:41.027645    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:15:42.374819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.374781    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"]
Oct 30 21:15:42.377955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.377933    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"
Oct 30 21:15:42.392879 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.392854    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa291d7-744e-4cac-811b-b6241e30f786-tmp\") pod \"jobset-operator-674cd59dc5-rbjxq\" (UID: \"6aa291d7-744e-4cac-811b-b6241e30f786\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"
Oct 30 21:15:42.392994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.392898    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5sl\" (UniqueName: \"kubernetes.io/projected/6aa291d7-744e-4cac-811b-b6241e30f786-kube-api-access-4j5sl\") pod \"jobset-operator-674cd59dc5-rbjxq\" (UID: \"6aa291d7-744e-4cac-811b-b6241e30f786\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"
Oct 30 21:15:42.403657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.403629    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:15:42.403886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.403865    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:15:42.404111 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.404090    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-c9jtd\""
Oct 30 21:15:42.406149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.406122    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"]
Oct 30 21:15:42.494240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.494203    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa291d7-744e-4cac-811b-b6241e30f786-tmp\") pod \"jobset-operator-674cd59dc5-rbjxq\" (UID: \"6aa291d7-744e-4cac-811b-b6241e30f786\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"
Oct 30 21:15:42.494417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.494253    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5sl\" (UniqueName: \"kubernetes.io/projected/6aa291d7-744e-4cac-811b-b6241e30f786-kube-api-access-4j5sl\") pod \"jobset-operator-674cd59dc5-rbjxq\" (UID: \"6aa291d7-744e-4cac-811b-b6241e30f786\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"
Oct 30 21:15:42.494610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.494590    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa291d7-744e-4cac-811b-b6241e30f786-tmp\") pod \"jobset-operator-674cd59dc5-rbjxq\" (UID: \"6aa291d7-744e-4cac-811b-b6241e30f786\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"
Oct 30 21:15:42.512861 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.512825    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5sl\" (UniqueName: \"kubernetes.io/projected/6aa291d7-744e-4cac-811b-b6241e30f786-kube-api-access-4j5sl\") pod \"jobset-operator-674cd59dc5-rbjxq\" (UID: \"6aa291d7-744e-4cac-811b-b6241e30f786\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"
Oct 30 21:15:42.702054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:42.702017    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"
Oct 30 21:15:43.007288 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:43.002260    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq"]
Oct 30 21:15:43.011025 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:15:43.010994    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6aa291d7_744e_4cac_811b_b6241e30f786.slice/crio-ffe3e1b8428174dcfaa95d38e00891b503419552920423c8b41121b12f12ee37 WatchSource:0}: Error finding container ffe3e1b8428174dcfaa95d38e00891b503419552920423c8b41121b12f12ee37: Status 404 returned error can't find the container with id ffe3e1b8428174dcfaa95d38e00891b503419552920423c8b41121b12f12ee37
Oct 30 21:15:43.644218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:43.644179    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq" event={"ID":"6aa291d7-744e-4cac-811b-b6241e30f786","Type":"ContainerStarted","Data":"ffe3e1b8428174dcfaa95d38e00891b503419552920423c8b41121b12f12ee37"}
Oct 30 21:15:46.660173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:46.660126    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq" event={"ID":"6aa291d7-744e-4cac-811b-b6241e30f786","Type":"ContainerStarted","Data":"3306cb9612e225e7db71e14ad3b90e7b977081a2fa4f1b17757f40e28e00bbd1"}
Oct 30 21:15:46.709827 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:46.709770    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rbjxq" podStartSLOduration=2.0605682 podStartE2EDuration="4.709753204s" totalImagesPullingTime="2.649185004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:46.709025891 +0000 UTC m=+346.838277348" watchObservedRunningTime="2026-10-30 21:15:46.709753204 +0000 UTC m=+346.839004664"
Oct 30 21:15:48.064039 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.064007    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"]
Oct 30 21:15:48.067752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.067729    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"
Oct 30 21:15:48.085250 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.085223    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 21:15:48.085362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.085252    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cdwnj\""
Oct 30 21:15:48.085362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.085253    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:15:48.085362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.085289    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:15:48.098323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.098301    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"]
Oct 30 21:15:48.139380 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.139345    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6fh\" (UniqueName: \"kubernetes.io/projected/ab559120-5838-4b2d-9a35-c9c9d62c0dd7-kube-api-access-nn6fh\") pod \"dns-operator-controller-manager-8557cbdb84-cc7x7\" (UID: \"ab559120-5838-4b2d-9a35-c9c9d62c0dd7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"
Oct 30 21:15:48.240263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.240223    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6fh\" (UniqueName: \"kubernetes.io/projected/ab559120-5838-4b2d-9a35-c9c9d62c0dd7-kube-api-access-nn6fh\") pod \"dns-operator-controller-manager-8557cbdb84-cc7x7\" (UID: \"ab559120-5838-4b2d-9a35-c9c9d62c0dd7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"
Oct 30 21:15:48.258939 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.258915    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6fh\" (UniqueName: \"kubernetes.io/projected/ab559120-5838-4b2d-9a35-c9c9d62c0dd7-kube-api-access-nn6fh\") pod \"dns-operator-controller-manager-8557cbdb84-cc7x7\" (UID: \"ab559120-5838-4b2d-9a35-c9c9d62c0dd7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"
Oct 30 21:15:48.379796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.379719    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"
Oct 30 21:15:48.626267 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.626232    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-864fdc7bd9-bpjf7"
Oct 30 21:15:48.680421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:48.680388    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"]
Oct 30 21:15:48.685759 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:15:48.685730    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab559120_5838_4b2d_9a35_c9c9d62c0dd7.slice/crio-2a7dc5143fbc4793adbf70628b986c5db652f508b5832b3bb17a2a2c47595711 WatchSource:0}: Error finding container 2a7dc5143fbc4793adbf70628b986c5db652f508b5832b3bb17a2a2c47595711: Status 404 returned error can't find the container with id 2a7dc5143fbc4793adbf70628b986c5db652f508b5832b3bb17a2a2c47595711
Oct 30 21:15:49.676909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:49.676869    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7" event={"ID":"ab559120-5838-4b2d-9a35-c9c9d62c0dd7","Type":"ContainerStarted","Data":"2a7dc5143fbc4793adbf70628b986c5db652f508b5832b3bb17a2a2c47595711"}
Oct 30 21:15:51.687215 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:51.687181    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7" event={"ID":"ab559120-5838-4b2d-9a35-c9c9d62c0dd7","Type":"ContainerStarted","Data":"2053fb1a9e6ed5ce707cc64a75f776accbe966e6a9491f6c58c991b01ef46634"}
Oct 30 21:15:51.687631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:51.687239    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"
Oct 30 21:15:51.733090 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:51.733044    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7" podStartSLOduration=1.134118115 podStartE2EDuration="3.733031084s" totalImagesPullingTime="2.598912969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:51.732557158 +0000 UTC m=+351.861808618" watchObservedRunningTime="2026-10-30 21:15:51.733031084 +0000 UTC m=+351.862282541"
Oct 30 21:15:52.511411 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:52.511372    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"]
Oct 30 21:15:52.514850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:52.514830    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"
Oct 30 21:15:52.543182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:52.543144    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4tp4n\""
Oct 30 21:15:52.544761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:52.544737    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"]
Oct 30 21:15:52.577917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:52.577886    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdjf\" (UniqueName: \"kubernetes.io/projected/6f413ea5-f172-43ed-bb8f-425675362ac7-kube-api-access-bmdjf\") pod \"limitador-operator-controller-manager-5778bb4449-q7jpk\" (UID: \"6f413ea5-f172-43ed-bb8f-425675362ac7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"
Oct 30 21:15:52.678632 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:52.678601    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdjf\" (UniqueName: \"kubernetes.io/projected/6f413ea5-f172-43ed-bb8f-425675362ac7-kube-api-access-bmdjf\") pod \"limitador-operator-controller-manager-5778bb4449-q7jpk\" (UID: \"6f413ea5-f172-43ed-bb8f-425675362ac7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"
Oct 30 21:15:52.697016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:52.696993    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdjf\" (UniqueName: \"kubernetes.io/projected/6f413ea5-f172-43ed-bb8f-425675362ac7-kube-api-access-bmdjf\") pod \"limitador-operator-controller-manager-5778bb4449-q7jpk\" (UID: \"6f413ea5-f172-43ed-bb8f-425675362ac7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"
Oct 30 21:15:52.826422 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:52.826346    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"
Oct 30 21:15:53.120605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:53.120519    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"]
Oct 30 21:15:53.125295 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:15:53.125263    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f413ea5_f172_43ed_bb8f_425675362ac7.slice/crio-164b95c3026598dda4a0b00d0274c061fefb0b40c0038944de8380975fb85d88 WatchSource:0}: Error finding container 164b95c3026598dda4a0b00d0274c061fefb0b40c0038944de8380975fb85d88: Status 404 returned error can't find the container with id 164b95c3026598dda4a0b00d0274c061fefb0b40c0038944de8380975fb85d88
Oct 30 21:15:53.696633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:53.696598    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk" event={"ID":"6f413ea5-f172-43ed-bb8f-425675362ac7","Type":"ContainerStarted","Data":"164b95c3026598dda4a0b00d0274c061fefb0b40c0038944de8380975fb85d88"}
Oct 30 21:15:54.702939 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:54.702899    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk" event={"ID":"6f413ea5-f172-43ed-bb8f-425675362ac7","Type":"ContainerStarted","Data":"e0e0dc5914c99b3f774ba634d1719d23c97eece823ecf55013b488ba5c20beb9"}
Oct 30 21:15:54.703425 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:54.702978    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"
Oct 30 21:15:54.743615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:54.743558    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk" podStartSLOduration=1.374715968 podStartE2EDuration="2.74354061s" totalImagesPullingTime="1.368824642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:54.742180908 +0000 UTC m=+354.871432359" watchObservedRunningTime="2026-10-30 21:15:54.74354061 +0000 UTC m=+354.872792069"
Oct 30 21:15:55.628169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.628126    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"]
Oct 30 21:15:55.631024 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.631001    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.646035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.646018    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 21:15:55.646410 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.646392    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 21:15:55.656490 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.656469    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6hdk6\""
Oct 30 21:15:55.657750 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.657729    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"]
Oct 30 21:15:55.701326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.701291    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4n92\" (UniqueName: \"kubernetes.io/projected/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-kube-api-access-f4n92\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.701500 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.701357    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.701500 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.701488    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.802083 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.802046    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.802570 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.802112    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.802570 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.802146    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4n92\" (UniqueName: \"kubernetes.io/projected/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-kube-api-access-f4n92\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.802723 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.802700    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.804504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.804473    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.820632 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.820605    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4n92\" (UniqueName: \"kubernetes.io/projected/e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83-kube-api-access-f4n92\") pod \"kuadrant-console-plugin-66c4fb575d-9snpv\" (UID: \"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:55.942502 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:55.942455    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"
Oct 30 21:15:56.241532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:56.241455    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv"]
Oct 30 21:15:56.246530 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:15:56.246498    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8d9fcc9_ca3d_47f4_a3ed_fa04f5215c83.slice/crio-b249c1c9ebda03add96475c52cbceeed53afd0902f35defaf4d70eca73adaf35 WatchSource:0}: Error finding container b249c1c9ebda03add96475c52cbceeed53afd0902f35defaf4d70eca73adaf35: Status 404 returned error can't find the container with id b249c1c9ebda03add96475c52cbceeed53afd0902f35defaf4d70eca73adaf35
Oct 30 21:15:56.712122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:56.712090    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv" event={"ID":"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83","Type":"ContainerStarted","Data":"b249c1c9ebda03add96475c52cbceeed53afd0902f35defaf4d70eca73adaf35"}
Oct 30 21:15:59.471648 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.471615    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"]
Oct 30 21:15:59.489365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.489329    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:15:59.502871 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.502835    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"]
Oct 30 21:15:59.536852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.536815    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l748\" (UniqueName: \"kubernetes.io/projected/71550ced-41dd-4cab-adf5-ac8cee060005-kube-api-access-7l748\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:15:59.537033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.536889    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71550ced-41dd-4cab-adf5-ac8cee060005-tmp\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:15:59.537033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.537001    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71550ced-41dd-4cab-adf5-ac8cee060005-tls-certs\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:15:59.545539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.545509    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:15:59.546534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.546514    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:15:59.546633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.546599    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pw6dq\""
Oct 30 21:15:59.637876 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.637834    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71550ced-41dd-4cab-adf5-ac8cee060005-tmp\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:15:59.638062 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.637885    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71550ced-41dd-4cab-adf5-ac8cee060005-tls-certs\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:15:59.638062 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.637928    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l748\" (UniqueName: \"kubernetes.io/projected/71550ced-41dd-4cab-adf5-ac8cee060005-kube-api-access-7l748\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:15:59.638062 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:15:59.638037    5886 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Oct 30 21:15:59.638511 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:15:59.638114    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71550ced-41dd-4cab-adf5-ac8cee060005-tls-certs podName:71550ced-41dd-4cab-adf5-ac8cee060005 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:00.138093498 +0000 UTC m=+360.267344946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/71550ced-41dd-4cab-adf5-ac8cee060005-tls-certs") pod "kube-auth-proxy-6fccbd9b79-fzlvq" (UID: "71550ced-41dd-4cab-adf5-ac8cee060005") : secret "kube-auth-proxy-tls" not found
Oct 30 21:15:59.640463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.640436    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71550ced-41dd-4cab-adf5-ac8cee060005-tmp\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:15:59.664084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:15:59.664050    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l748\" (UniqueName: \"kubernetes.io/projected/71550ced-41dd-4cab-adf5-ac8cee060005-kube-api-access-7l748\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:16:00.143642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:00.143604    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71550ced-41dd-4cab-adf5-ac8cee060005-tls-certs\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:16:00.153364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:00.153334    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71550ced-41dd-4cab-adf5-ac8cee060005-tls-certs\") pod \"kube-auth-proxy-6fccbd9b79-fzlvq\" (UID: \"71550ced-41dd-4cab-adf5-ac8cee060005\") " pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:16:00.418612 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:00.418579    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pw6dq\""
Oct 30 21:16:00.426764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:00.426737    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"
Oct 30 21:16:00.737211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:00.737119    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq"]
Oct 30 21:16:00.743128 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:16:00.743094    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71550ced_41dd_4cab_adf5_ac8cee060005.slice/crio-346ee81a9dec09af46d8aafceabfc6edb0ca39a5ea39c87872f16d9dd0691b62 WatchSource:0}: Error finding container 346ee81a9dec09af46d8aafceabfc6edb0ca39a5ea39c87872f16d9dd0691b62: Status 404 returned error can't find the container with id 346ee81a9dec09af46d8aafceabfc6edb0ca39a5ea39c87872f16d9dd0691b62
Oct 30 21:16:01.744203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:01.744140    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv" event={"ID":"e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83","Type":"ContainerStarted","Data":"0bd89f2c8aee12c467adf177d7fb4ed3ff3807a9b8d16bbe793d83dd4443c8c2"}
Oct 30 21:16:01.746962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:01.746905    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq" event={"ID":"71550ced-41dd-4cab-adf5-ac8cee060005","Type":"ContainerStarted","Data":"346ee81a9dec09af46d8aafceabfc6edb0ca39a5ea39c87872f16d9dd0691b62"}
Oct 30 21:16:01.780995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:01.780947    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-9snpv" podStartSLOduration=1.662719916 podStartE2EDuration="6.780934363s" totalImagesPullingTime="5.118214447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:01.779959675 +0000 UTC m=+361.909211133" watchObservedRunningTime="2026-10-30 21:16:01.780934363 +0000 UTC m=+361.910185820"
Oct 30 21:16:02.694443 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:02.694410    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cc7x7"
Oct 30 21:16:03.111094 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:03.111068    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:16:03.767607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:03.767559    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq" event={"ID":"71550ced-41dd-4cab-adf5-ac8cee060005","Type":"ContainerStarted","Data":"fb4a22690b21ced4587d04cea45536b1325f0c069d4eeba49b059eb767e01398"}
Oct 30 21:16:03.809221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:03.809142    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fccbd9b79-fzlvq" podStartSLOduration=2.456542908 podStartE2EDuration="4.809130451s" totalImagesPullingTime="2.352587543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:03.808491028 +0000 UTC m=+363.937742489" watchObservedRunningTime="2026-10-30 21:16:03.809130451 +0000 UTC m=+363.938381908"
Oct 30 21:16:04.153788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.153754    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"]
Oct 30 21:16:04.157353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.157330    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:04.172472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.172446    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rqzpl\""
Oct 30 21:16:04.185096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.185072    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"]
Oct 30 21:16:04.280773 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.280739    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gwgz\" (UniqueName: \"kubernetes.io/projected/914c9e20-a98b-4655-8405-7dfbfc030564-kube-api-access-4gwgz\") pod \"kuadrant-operator-controller-manager-678d87766f-q6j5d\" (UID: \"914c9e20-a98b-4655-8405-7dfbfc030564\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:04.280957 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.280865    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/914c9e20-a98b-4655-8405-7dfbfc030564-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q6j5d\" (UID: \"914c9e20-a98b-4655-8405-7dfbfc030564\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:04.381954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.381916    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/914c9e20-a98b-4655-8405-7dfbfc030564-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q6j5d\" (UID: \"914c9e20-a98b-4655-8405-7dfbfc030564\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:04.382139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.381976    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gwgz\" (UniqueName: \"kubernetes.io/projected/914c9e20-a98b-4655-8405-7dfbfc030564-kube-api-access-4gwgz\") pod \"kuadrant-operator-controller-manager-678d87766f-q6j5d\" (UID: \"914c9e20-a98b-4655-8405-7dfbfc030564\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:04.382375 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.382352    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/914c9e20-a98b-4655-8405-7dfbfc030564-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-q6j5d\" (UID: \"914c9e20-a98b-4655-8405-7dfbfc030564\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:04.401776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.401747    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gwgz\" (UniqueName: \"kubernetes.io/projected/914c9e20-a98b-4655-8405-7dfbfc030564-kube-api-access-4gwgz\") pod \"kuadrant-operator-controller-manager-678d87766f-q6j5d\" (UID: \"914c9e20-a98b-4655-8405-7dfbfc030564\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:04.469598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.469519    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:04.798816 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:04.798728    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"]
Oct 30 21:16:04.804009 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:16:04.803981    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914c9e20_a98b_4655_8405_7dfbfc030564.slice/crio-2a7f25d1a30167a6551459bbf77f58cea5113e0be6f1a7dcf687813df1cf70eb WatchSource:0}: Error finding container 2a7f25d1a30167a6551459bbf77f58cea5113e0be6f1a7dcf687813df1cf70eb: Status 404 returned error can't find the container with id 2a7f25d1a30167a6551459bbf77f58cea5113e0be6f1a7dcf687813df1cf70eb
Oct 30 21:16:05.708966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:05.708931    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q7jpk"
Oct 30 21:16:05.780379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:05.780345    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d" event={"ID":"914c9e20-a98b-4655-8405-7dfbfc030564","Type":"ContainerStarted","Data":"2a7f25d1a30167a6551459bbf77f58cea5113e0be6f1a7dcf687813df1cf70eb"}
Oct 30 21:16:09.802415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:09.802321    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d" event={"ID":"914c9e20-a98b-4655-8405-7dfbfc030564","Type":"ContainerStarted","Data":"50b7f82d4cd32163619fff320e1680b469212c9eca6ebd1b17e6be39e222c7f1"}
Oct 30 21:16:09.802415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:09.802385    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:09.841895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:09.841842    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d" podStartSLOduration=1.116649252 podStartE2EDuration="5.841828079s" totalImagesPullingTime="4.725178827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:09.840401436 +0000 UTC m=+369.969652895" watchObservedRunningTime="2026-10-30 21:16:09.841828079 +0000 UTC m=+369.971079528"
Oct 30 21:16:11.027656 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:16:11.027575    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:16:20.809078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:16:20.809044    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-q6j5d"
Oct 30 21:16:41.027150 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:16:41.027106    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:17:03.727777 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.727741    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-fmm47"]
Oct 30 21:17:03.732356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.732334    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:03.749183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.749144    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:17:03.752993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.752977    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:17:03.759413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.759390    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgwbf\" (UniqueName: \"kubernetes.io/projected/584aa75a-57b4-40a0-98c9-dee63e23b032-kube-api-access-fgwbf\") pod \"kserve-controller-manager-776cc66d47-fmm47\" (UID: \"584aa75a-57b4-40a0-98c9-dee63e23b032\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:03.759528 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.759425    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/584aa75a-57b4-40a0-98c9-dee63e23b032-cert\") pod \"kserve-controller-manager-776cc66d47-fmm47\" (UID: \"584aa75a-57b4-40a0-98c9-dee63e23b032\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:03.762246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.762222    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-fmm47"]
Oct 30 21:17:03.764026 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.764009    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 21:17:03.764088 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.764009    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rrfk4\""
Oct 30 21:17:03.817397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.817355    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-qw5z9"]
Oct 30 21:17:03.821646 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.821627    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:03.840623 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.840594    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 21:17:03.842257 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.842231    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5n7dj\""
Oct 30 21:17:03.848885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.848860    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-qw5z9"]
Oct 30 21:17:03.859969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.859945    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-ltdxn"]
Oct 30 21:17:03.860244 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.860224    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/584aa75a-57b4-40a0-98c9-dee63e23b032-cert\") pod \"kserve-controller-manager-776cc66d47-fmm47\" (UID: \"584aa75a-57b4-40a0-98c9-dee63e23b032\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:03.860394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.860378    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57sg\" (UniqueName: \"kubernetes.io/projected/2d5eaaf1-15a5-4663-a680-4eb2107b1f75-kube-api-access-j57sg\") pod \"model-serving-api-75bd89fdfc-qw5z9\" (UID: \"2d5eaaf1-15a5-4663-a680-4eb2107b1f75\") " pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:03.860434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.860409    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d5eaaf1-15a5-4663-a680-4eb2107b1f75-tls-certs\") pod \"model-serving-api-75bd89fdfc-qw5z9\" (UID: \"2d5eaaf1-15a5-4663-a680-4eb2107b1f75\") " pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:03.860487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.860439    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgwbf\" (UniqueName: \"kubernetes.io/projected/584aa75a-57b4-40a0-98c9-dee63e23b032-kube-api-access-fgwbf\") pod \"kserve-controller-manager-776cc66d47-fmm47\" (UID: \"584aa75a-57b4-40a0-98c9-dee63e23b032\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:03.863015 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.862988    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/584aa75a-57b4-40a0-98c9-dee63e23b032-cert\") pod \"kserve-controller-manager-776cc66d47-fmm47\" (UID: \"584aa75a-57b4-40a0-98c9-dee63e23b032\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:03.863365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.863350    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:03.878665 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.878645    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jvcwx\""
Oct 30 21:17:03.878787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.878679    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 21:17:03.878787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.878709    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 21:17:03.881783 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.881760    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgwbf\" (UniqueName: \"kubernetes.io/projected/584aa75a-57b4-40a0-98c9-dee63e23b032-kube-api-access-fgwbf\") pod \"kserve-controller-manager-776cc66d47-fmm47\" (UID: \"584aa75a-57b4-40a0-98c9-dee63e23b032\") " pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:03.883427 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.883403    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ltdxn"]
Oct 30 21:17:03.961498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.961459    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12837eab-8ef2-4a9a-8423-fd13972fb1ad-cert\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:03.961674 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.961512    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cd7\" (UniqueName: \"kubernetes.io/projected/12837eab-8ef2-4a9a-8423-fd13972fb1ad-kube-api-access-t5cd7\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:03.961674 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.961632    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57sg\" (UniqueName: \"kubernetes.io/projected/2d5eaaf1-15a5-4663-a680-4eb2107b1f75-kube-api-access-j57sg\") pod \"model-serving-api-75bd89fdfc-qw5z9\" (UID: \"2d5eaaf1-15a5-4663-a680-4eb2107b1f75\") " pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:03.961674 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.961664    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d5eaaf1-15a5-4663-a680-4eb2107b1f75-tls-certs\") pod \"model-serving-api-75bd89fdfc-qw5z9\" (UID: \"2d5eaaf1-15a5-4663-a680-4eb2107b1f75\") " pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:03.961800 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.961697    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12837eab-8ef2-4a9a-8423-fd13972fb1ad-metrics-certs\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:03.964145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.964124    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d5eaaf1-15a5-4663-a680-4eb2107b1f75-tls-certs\") pod \"model-serving-api-75bd89fdfc-qw5z9\" (UID: \"2d5eaaf1-15a5-4663-a680-4eb2107b1f75\") " pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:03.983633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:03.983534    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57sg\" (UniqueName: \"kubernetes.io/projected/2d5eaaf1-15a5-4663-a680-4eb2107b1f75-kube-api-access-j57sg\") pod \"model-serving-api-75bd89fdfc-qw5z9\" (UID: \"2d5eaaf1-15a5-4663-a680-4eb2107b1f75\") " pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:04.044478 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.044445    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:04.062694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.062661    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12837eab-8ef2-4a9a-8423-fd13972fb1ad-metrics-certs\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:04.062807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.062713    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12837eab-8ef2-4a9a-8423-fd13972fb1ad-cert\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:04.062807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.062780    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cd7\" (UniqueName: \"kubernetes.io/projected/12837eab-8ef2-4a9a-8423-fd13972fb1ad-kube-api-access-t5cd7\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:04.065072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.065048    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12837eab-8ef2-4a9a-8423-fd13972fb1ad-metrics-certs\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:04.065185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.065087    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12837eab-8ef2-4a9a-8423-fd13972fb1ad-cert\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:04.081125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.081086    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cd7\" (UniqueName: \"kubernetes.io/projected/12837eab-8ef2-4a9a-8423-fd13972fb1ad-kube-api-access-t5cd7\") pod \"odh-model-controller-769875db56-ltdxn\" (UID: \"12837eab-8ef2-4a9a-8423-fd13972fb1ad\") " pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:04.135504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.135474    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:04.182246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.180779    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:04.336957 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.336922    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-fmm47"]
Oct 30 21:17:04.341554 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:17:04.341526    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584aa75a_57b4_40a0_98c9_dee63e23b032.slice/crio-344ea3deb6642e1c205ed9681d4c7311d56a61b6a63fd7f6f22c9288973b3f08 WatchSource:0}: Error finding container 344ea3deb6642e1c205ed9681d4c7311d56a61b6a63fd7f6f22c9288973b3f08: Status 404 returned error can't find the container with id 344ea3deb6642e1c205ed9681d4c7311d56a61b6a63fd7f6f22c9288973b3f08
Oct 30 21:17:04.428013 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.427977    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-qw5z9"]
Oct 30 21:17:04.433221 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:17:04.433192    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5eaaf1_15a5_4663_a680_4eb2107b1f75.slice/crio-89b05bf7779dcbb552b0c912b9358b13a996a57074a5ed56ef7d9557ba6bae9b WatchSource:0}: Error finding container 89b05bf7779dcbb552b0c912b9358b13a996a57074a5ed56ef7d9557ba6bae9b: Status 404 returned error can't find the container with id 89b05bf7779dcbb552b0c912b9358b13a996a57074a5ed56ef7d9557ba6bae9b
Oct 30 21:17:04.501083 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:04.501024    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ltdxn"]
Oct 30 21:17:04.505798 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:17:04.505773    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12837eab_8ef2_4a9a_8423_fd13972fb1ad.slice/crio-901e1fd41a8f2d93c7a946ccfe351ac36a16afaeb5cc2a76df5954f2b6611c70 WatchSource:0}: Error finding container 901e1fd41a8f2d93c7a946ccfe351ac36a16afaeb5cc2a76df5954f2b6611c70: Status 404 returned error can't find the container with id 901e1fd41a8f2d93c7a946ccfe351ac36a16afaeb5cc2a76df5954f2b6611c70
Oct 30 21:17:05.059651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:05.059510    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ltdxn" event={"ID":"12837eab-8ef2-4a9a-8423-fd13972fb1ad","Type":"ContainerStarted","Data":"901e1fd41a8f2d93c7a946ccfe351ac36a16afaeb5cc2a76df5954f2b6611c70"}
Oct 30 21:17:05.062348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:05.062314    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9" event={"ID":"2d5eaaf1-15a5-4663-a680-4eb2107b1f75","Type":"ContainerStarted","Data":"89b05bf7779dcbb552b0c912b9358b13a996a57074a5ed56ef7d9557ba6bae9b"}
Oct 30 21:17:05.066975 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:05.066946    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47" event={"ID":"584aa75a-57b4-40a0-98c9-dee63e23b032","Type":"ContainerStarted","Data":"344ea3deb6642e1c205ed9681d4c7311d56a61b6a63fd7f6f22c9288973b3f08"}
Oct 30 21:17:07.081727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:07.081688    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9" event={"ID":"2d5eaaf1-15a5-4663-a680-4eb2107b1f75","Type":"ContainerStarted","Data":"20c140b5cbf3043b437df1669ae244a6e2b28a0be19bc8b3ffdb6b4c0d09240a"}
Oct 30 21:17:07.082110 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:07.081792    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:07.084224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:07.083847    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47" event={"ID":"584aa75a-57b4-40a0-98c9-dee63e23b032","Type":"ContainerStarted","Data":"5d1fa6bdfd1a52ca8043064269813da23444b58eab3675402986d6d455cf7339"}
Oct 30 21:17:07.084224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:07.083950    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:07.119638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:07.119560    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9" podStartSLOduration=1.7996947479999998 podStartE2EDuration="4.119542692s" totalImagesPullingTime="2.319847944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:07.118124747 +0000 UTC m=+427.247376206" watchObservedRunningTime="2026-10-30 21:17:07.119542692 +0000 UTC m=+427.248794149"
Oct 30 21:17:07.156883 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:07.156486    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47" podStartSLOduration=1.507281216 podStartE2EDuration="4.156468787s" totalImagesPullingTime="2.649187571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:07.154799945 +0000 UTC m=+427.284051401" watchObservedRunningTime="2026-10-30 21:17:07.156468787 +0000 UTC m=+427.285720243"
Oct 30 21:17:08.090468 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:08.090430    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ltdxn" event={"ID":"12837eab-8ef2-4a9a-8423-fd13972fb1ad","Type":"ContainerStarted","Data":"a20c490e59d01e8d7aecf320ce5673cfe331713d2d66040ba7e1845ea433449d"}
Oct 30 21:17:08.129224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:08.129134    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-ltdxn" podStartSLOduration=2.099836983 podStartE2EDuration="5.12911701s" totalImagesPullingTime="3.029280027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:08.12777815 +0000 UTC m=+428.257029606" watchObservedRunningTime="2026-10-30 21:17:08.12911701 +0000 UTC m=+428.258368468"
Oct 30 21:17:09.096249 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:09.096213    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:11.027956 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:17:11.027920    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:17:18.094592 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:18.094558    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-qw5z9"
Oct 30 21:17:20.102877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:20.102846    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-ltdxn"
Oct 30 21:17:38.096240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:17:38.096206    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-fmm47"
Oct 30 21:17:41.027877 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:17:41.027801    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:18:00.341432 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.341394    5886 generic.go:356] "Generic (PLEG): container finished" podID="f9457827-beb1-4755-bb94-29ac9c2c1659" containerID="5cef17eb5485a66ac428b90fa63008b4b093994a6351446613bad310a99a40bd" exitCode=0
Oct 30 21:18:00.341926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.341506    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" event={"ID":"f9457827-beb1-4755-bb94-29ac9c2c1659","Type":"ContainerDied","Data":"5cef17eb5485a66ac428b90fa63008b4b093994a6351446613bad310a99a40bd"}
Oct 30 21:18:00.342007 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.341931    5886 scope.go:122] "RemoveContainer" containerID="5cef17eb5485a66ac428b90fa63008b4b093994a6351446613bad310a99a40bd"
Oct 30 21:18:00.343096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.343062    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:00.343225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.343148    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:00.343225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.343201    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:18:00.445351 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.445123    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"]
Oct 30 21:18:00.448172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.448139    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.464369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.461911    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:00.464369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.461923    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:00.464501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.464476    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:18:00.464587 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.464571    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 21:18:00.464675 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.464660    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q645h\""
Oct 30 21:18:00.475254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.475219    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"]
Oct 30 21:18:00.548350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.548311    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qcvj\" (UniqueName: \"kubernetes.io/projected/729b7547-83d6-4451-bde2-c54e7d019008-kube-api-access-9qcvj\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.548505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.548360    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/729b7547-83d6-4451-bde2-c54e7d019008-openshift-service-ca\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.548505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.548475    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/729b7547-83d6-4451-bde2-c54e7d019008-maas-api-tls\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.548581 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.548529    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729b7547-83d6-4451-bde2-c54e7d019008-metrics-certs\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.649637 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.649599    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/729b7547-83d6-4451-bde2-c54e7d019008-maas-api-tls\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.649796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.649668    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729b7547-83d6-4451-bde2-c54e7d019008-metrics-certs\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.649796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.649715    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qcvj\" (UniqueName: \"kubernetes.io/projected/729b7547-83d6-4451-bde2-c54e7d019008-kube-api-access-9qcvj\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.649796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.649741    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/729b7547-83d6-4451-bde2-c54e7d019008-openshift-service-ca\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.650501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.650467    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/729b7547-83d6-4451-bde2-c54e7d019008-openshift-service-ca\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.652131 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.652108    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/729b7547-83d6-4451-bde2-c54e7d019008-maas-api-tls\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.652250 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.652174    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/729b7547-83d6-4451-bde2-c54e7d019008-metrics-certs\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.673897 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.673871    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qcvj\" (UniqueName: \"kubernetes.io/projected/729b7547-83d6-4451-bde2-c54e7d019008-kube-api-access-9qcvj\") pod \"maas-api-6b54f57f95-gs564\" (UID: \"729b7547-83d6-4451-bde2-c54e7d019008\") " pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:00.766240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:00.765337    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:01.080235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:01.075104    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"]
Oct 30 21:18:01.089379 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:18:01.089342    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729b7547_83d6_4451_bde2_c54e7d019008.slice/crio-20532b66a56d19e26e8a38b8e3883dbcebf7815ee2e34ae5de6d3c13ec833713 WatchSource:0}: Error finding container 20532b66a56d19e26e8a38b8e3883dbcebf7815ee2e34ae5de6d3c13ec833713: Status 404 returned error can't find the container with id 20532b66a56d19e26e8a38b8e3883dbcebf7815ee2e34ae5de6d3c13ec833713
Oct 30 21:18:01.350610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:01.350522    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564" event={"ID":"729b7547-83d6-4451-bde2-c54e7d019008","Type":"ContainerStarted","Data":"20532b66a56d19e26e8a38b8e3883dbcebf7815ee2e34ae5de6d3c13ec833713"}
Oct 30 21:18:01.352570 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:01.352543    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" event={"ID":"f9457827-beb1-4755-bb94-29ac9c2c1659","Type":"ContainerStarted","Data":"e79b1c170fc8c6df875ed476b064ad01f074c3fb21bf2e194caf7ba2058bb4c9"}
Oct 30 21:18:01.821733 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:01.821702    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:18:01.823064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:01.822758    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cf7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 21:18:01.823064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:01.822820    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" podUID="f9457827-beb1-4755-bb94-29ac9c2c1659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 21:18:02.821379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:02.821341    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cf7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 21:18:02.821847 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:02.821394    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" podUID="f9457827-beb1-4755-bb94-29ac9c2c1659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 21:18:03.821279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:03.821239    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cf7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 21:18:03.821477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:03.821288    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" podUID="f9457827-beb1-4755-bb94-29ac9c2c1659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 21:18:04.395378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:04.395339    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564" event={"ID":"729b7547-83d6-4451-bde2-c54e7d019008","Type":"ContainerStarted","Data":"f4465262b6b2ed50e2b016b1b7d7929fde6d0a35202027faee44a567bd211e40"}
Oct 30 21:18:04.396033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:04.396013    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:04.435603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:04.435543    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564" podStartSLOduration=1.313927379 podStartE2EDuration="4.435529371s" totalImagesPullingTime="3.121601992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:04.433802613 +0000 UTC m=+484.563054069" watchObservedRunningTime="2026-10-30 21:18:04.435529371 +0000 UTC m=+484.564780844"
Oct 30 21:18:04.821217 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:04.821149    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cf7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 21:18:04.821394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:04.821233    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" podUID="f9457827-beb1-4755-bb94-29ac9c2c1659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 21:18:05.221351 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:05.221314    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:18:05.820702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:05.820667    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cf7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Oct 30 21:18:05.820877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:05.820717    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb" podUID="f9457827-beb1-4755-bb94-29ac9c2c1659" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Oct 30 21:18:06.824984 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:06.824954    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:18:06.825997 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:06.825974    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cf7lb"
Oct 30 21:18:11.027443 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:18:11.027408    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:18:11.410659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:11.410634    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b54f57f95-gs564"
Oct 30 21:18:37.562264 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.562229    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jwhvc"]
Oct 30 21:18:37.570215 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.570188    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:37.583674 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.583648    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jwhvc"]
Oct 30 21:18:37.593616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.593581    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 21:18:37.595725 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.595704    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 21:18:37.595803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.595770    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-ztgcs\""
Oct 30 21:18:37.697206 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.697142    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndbk\" (UniqueName: \"kubernetes.io/projected/780269a2-a9c7-4377-9b8a-be67df7871ca-kube-api-access-6ndbk\") pod \"ai-gateway-controller-7d85675f98-jwhvc\" (UID: \"780269a2-a9c7-4377-9b8a-be67df7871ca\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:37.697371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.697210    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780269a2-a9c7-4377-9b8a-be67df7871ca-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-jwhvc\" (UID: \"780269a2-a9c7-4377-9b8a-be67df7871ca\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:37.798353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.798322    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndbk\" (UniqueName: \"kubernetes.io/projected/780269a2-a9c7-4377-9b8a-be67df7871ca-kube-api-access-6ndbk\") pod \"ai-gateway-controller-7d85675f98-jwhvc\" (UID: \"780269a2-a9c7-4377-9b8a-be67df7871ca\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:37.798353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.798356    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780269a2-a9c7-4377-9b8a-be67df7871ca-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-jwhvc\" (UID: \"780269a2-a9c7-4377-9b8a-be67df7871ca\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:37.800712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.800683    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780269a2-a9c7-4377-9b8a-be67df7871ca-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-jwhvc\" (UID: \"780269a2-a9c7-4377-9b8a-be67df7871ca\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:37.817144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.817089    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndbk\" (UniqueName: \"kubernetes.io/projected/780269a2-a9c7-4377-9b8a-be67df7871ca-kube-api-access-6ndbk\") pod \"ai-gateway-controller-7d85675f98-jwhvc\" (UID: \"780269a2-a9c7-4377-9b8a-be67df7871ca\") " pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:37.882938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:37.882894    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:38.176171 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:38.176127    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jwhvc"]
Oct 30 21:18:38.181386 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:18:38.181354    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780269a2_a9c7_4377_9b8a_be67df7871ca.slice/crio-c2654f54ac3fdb4274a5326226c9c469d98977f84add23cebdc3eeef147d9df4 WatchSource:0}: Error finding container c2654f54ac3fdb4274a5326226c9c469d98977f84add23cebdc3eeef147d9df4: Status 404 returned error can't find the container with id c2654f54ac3fdb4274a5326226c9c469d98977f84add23cebdc3eeef147d9df4
Oct 30 21:18:38.559120 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:38.559034    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc" event={"ID":"780269a2-a9c7-4377-9b8a-be67df7871ca","Type":"ContainerStarted","Data":"c2654f54ac3fdb4274a5326226c9c469d98977f84add23cebdc3eeef147d9df4"}
Oct 30 21:18:40.570420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.570376    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc" event={"ID":"780269a2-a9c7-4377-9b8a-be67df7871ca","Type":"ContainerStarted","Data":"1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2"}
Oct 30 21:18:40.570957 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.570474    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:40.606609 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.606555    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc" podStartSLOduration=2.11387024 podStartE2EDuration="3.606543203s" totalImagesPullingTime="1.492672963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:40.605348278 +0000 UTC m=+520.734599714" watchObservedRunningTime="2026-10-30 21:18:40.606543203 +0000 UTC m=+520.735794674"
Oct 30 21:18:40.785481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.785446    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"]
Oct 30 21:18:40.789134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.789114    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:40.807003 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.806977    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 21:18:40.808531 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.808509    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"]
Oct 30 21:18:40.811972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.811953    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"]
Oct 30 21:18:40.815495 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.815474    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:40.830176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.830134    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6vftd\""
Oct 30 21:18:40.841464 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.841434    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"]
Oct 30 21:18:40.924132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.924087    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee881cfc-ad3c-4017-8504-722b0543c67f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-6l9b4\" (UID: \"ee881cfc-ad3c-4017-8504-722b0543c67f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:40.924334 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.924137    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9120e132-28e6-49ac-a31e-d263a71cea20-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9x9cn\" (UID: \"9120e132-28e6-49ac-a31e-d263a71cea20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:40.924334 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.924241    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjnj\" (UniqueName: \"kubernetes.io/projected/ee881cfc-ad3c-4017-8504-722b0543c67f-kube-api-access-znjnj\") pod \"payload-processing-55fbbbf9dc-6l9b4\" (UID: \"ee881cfc-ad3c-4017-8504-722b0543c67f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:40.924334 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:40.924294    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km52v\" (UniqueName: \"kubernetes.io/projected/9120e132-28e6-49ac-a31e-d263a71cea20-kube-api-access-km52v\") pod \"payload-pre-processing-7b98bb678c-9x9cn\" (UID: \"9120e132-28e6-49ac-a31e-d263a71cea20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:41.025859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.025823    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km52v\" (UniqueName: \"kubernetes.io/projected/9120e132-28e6-49ac-a31e-d263a71cea20-kube-api-access-km52v\") pod \"payload-pre-processing-7b98bb678c-9x9cn\" (UID: \"9120e132-28e6-49ac-a31e-d263a71cea20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:41.026059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.025939    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee881cfc-ad3c-4017-8504-722b0543c67f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-6l9b4\" (UID: \"ee881cfc-ad3c-4017-8504-722b0543c67f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:41.026059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.025969    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9120e132-28e6-49ac-a31e-d263a71cea20-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9x9cn\" (UID: \"9120e132-28e6-49ac-a31e-d263a71cea20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:41.026059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.026007    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjnj\" (UniqueName: \"kubernetes.io/projected/ee881cfc-ad3c-4017-8504-722b0543c67f-kube-api-access-znjnj\") pod \"payload-processing-55fbbbf9dc-6l9b4\" (UID: \"ee881cfc-ad3c-4017-8504-722b0543c67f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:41.026749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.026722    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee881cfc-ad3c-4017-8504-722b0543c67f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-6l9b4\" (UID: \"ee881cfc-ad3c-4017-8504-722b0543c67f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:41.026897 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.026849    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9120e132-28e6-49ac-a31e-d263a71cea20-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9x9cn\" (UID: \"9120e132-28e6-49ac-a31e-d263a71cea20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:41.032811 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:18:41.032769    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:18:41.044390 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.044356    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km52v\" (UniqueName: \"kubernetes.io/projected/9120e132-28e6-49ac-a31e-d263a71cea20-kube-api-access-km52v\") pod \"payload-pre-processing-7b98bb678c-9x9cn\" (UID: \"9120e132-28e6-49ac-a31e-d263a71cea20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:41.044681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.044654    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjnj\" (UniqueName: \"kubernetes.io/projected/ee881cfc-ad3c-4017-8504-722b0543c67f-kube-api-access-znjnj\") pod \"payload-processing-55fbbbf9dc-6l9b4\" (UID: \"ee881cfc-ad3c-4017-8504-722b0543c67f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:41.102009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.101929    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:41.132462 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.132431    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:41.399033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.399008    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"]
Oct 30 21:18:41.404647 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:18:41.404623    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9120e132_28e6_49ac_a31e_d263a71cea20.slice/crio-722dc370a7b1c388220211d0c834dd3cb4096243f883c685e5762177350e78c2 WatchSource:0}: Error finding container 722dc370a7b1c388220211d0c834dd3cb4096243f883c685e5762177350e78c2: Status 404 returned error can't find the container with id 722dc370a7b1c388220211d0c834dd3cb4096243f883c685e5762177350e78c2
Oct 30 21:18:41.442144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.442108    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"]
Oct 30 21:18:41.448369 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:18:41.448337    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee881cfc_ad3c_4017_8504_722b0543c67f.slice/crio-d87d60f927dfe62a619b76d37505a7b731acd2e1b8008a5574890b8ccbc7ab0c WatchSource:0}: Error finding container d87d60f927dfe62a619b76d37505a7b731acd2e1b8008a5574890b8ccbc7ab0c: Status 404 returned error can't find the container with id d87d60f927dfe62a619b76d37505a7b731acd2e1b8008a5574890b8ccbc7ab0c
Oct 30 21:18:41.576346 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.576307    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4" event={"ID":"ee881cfc-ad3c-4017-8504-722b0543c67f","Type":"ContainerStarted","Data":"d87d60f927dfe62a619b76d37505a7b731acd2e1b8008a5574890b8ccbc7ab0c"}
Oct 30 21:18:41.577538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:41.577509    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn" event={"ID":"9120e132-28e6-49ac-a31e-d263a71cea20","Type":"ContainerStarted","Data":"722dc370a7b1c388220211d0c834dd3cb4096243f883c685e5762177350e78c2"}
Oct 30 21:18:44.596235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:44.596192    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4" event={"ID":"ee881cfc-ad3c-4017-8504-722b0543c67f","Type":"ContainerStarted","Data":"3447085d82400590106e5bdb04c96834cd55a88371c05a9c976e8b788dd036ea"}
Oct 30 21:18:44.596757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:44.596314    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:44.597769 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:44.597745    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn" event={"ID":"9120e132-28e6-49ac-a31e-d263a71cea20","Type":"ContainerStarted","Data":"87bb5d79530c34aa6f3a7b4575de4a321f0b4ca245c31bd49342d000aa9f4284"}
Oct 30 21:18:44.597884 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:44.597795    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:18:44.633055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:44.632996    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4" podStartSLOduration=2.369165588 podStartE2EDuration="4.632984005s" totalImagesPullingTime="2.263818417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:44.632114753 +0000 UTC m=+524.761366209" watchObservedRunningTime="2026-10-30 21:18:44.632984005 +0000 UTC m=+524.762235462"
Oct 30 21:18:44.667972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:44.667921    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn" podStartSLOduration=2.4577024339999998 podStartE2EDuration="4.667905534s" totalImagesPullingTime="2.2102031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:44.66688537 +0000 UTC m=+524.796136809" watchObservedRunningTime="2026-10-30 21:18:44.667905534 +0000 UTC m=+524.797157026"
Oct 30 21:18:51.579322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:51.579289    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:18:51.856538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:51.856454    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"]
Oct 30 21:18:51.860313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:51.860290    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:51.878028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:51.878001    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"]
Oct 30 21:18:52.026407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.026365    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee59b1b-5575-4096-98e1-1f78882e72dc-metrics-certs\") pod \"ai-gateway-controller-845f9f9b5f-ggr56\" (UID: \"0ee59b1b-5575-4096-98e1-1f78882e72dc\") " pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:52.026561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.026433    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99qf\" (UniqueName: \"kubernetes.io/projected/0ee59b1b-5575-4096-98e1-1f78882e72dc-kube-api-access-l99qf\") pod \"ai-gateway-controller-845f9f9b5f-ggr56\" (UID: \"0ee59b1b-5575-4096-98e1-1f78882e72dc\") " pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:52.127698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.127599    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee59b1b-5575-4096-98e1-1f78882e72dc-metrics-certs\") pod \"ai-gateway-controller-845f9f9b5f-ggr56\" (UID: \"0ee59b1b-5575-4096-98e1-1f78882e72dc\") " pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:52.127698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.127684    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99qf\" (UniqueName: \"kubernetes.io/projected/0ee59b1b-5575-4096-98e1-1f78882e72dc-kube-api-access-l99qf\") pod \"ai-gateway-controller-845f9f9b5f-ggr56\" (UID: \"0ee59b1b-5575-4096-98e1-1f78882e72dc\") " pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:52.130077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.130048    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee59b1b-5575-4096-98e1-1f78882e72dc-metrics-certs\") pod \"ai-gateway-controller-845f9f9b5f-ggr56\" (UID: \"0ee59b1b-5575-4096-98e1-1f78882e72dc\") " pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:52.146122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.146092    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99qf\" (UniqueName: \"kubernetes.io/projected/0ee59b1b-5575-4096-98e1-1f78882e72dc-kube-api-access-l99qf\") pod \"ai-gateway-controller-845f9f9b5f-ggr56\" (UID: \"0ee59b1b-5575-4096-98e1-1f78882e72dc\") " pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:52.172932 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.172911    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:52.464839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.464802    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"]
Oct 30 21:18:52.469908 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:18:52.469880    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee59b1b_5575_4096_98e1_1f78882e72dc.slice/crio-263729e949802c724e370e3c0bad0222dac49a6afe80a342acf5ec562257dea6 WatchSource:0}: Error finding container 263729e949802c724e370e3c0bad0222dac49a6afe80a342acf5ec562257dea6: Status 404 returned error can't find the container with id 263729e949802c724e370e3c0bad0222dac49a6afe80a342acf5ec562257dea6
Oct 30 21:18:52.639053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:52.639014    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56" event={"ID":"0ee59b1b-5575-4096-98e1-1f78882e72dc","Type":"ContainerStarted","Data":"263729e949802c724e370e3c0bad0222dac49a6afe80a342acf5ec562257dea6"}
Oct 30 21:18:53.645017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:53.644978    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56" event={"ID":"0ee59b1b-5575-4096-98e1-1f78882e72dc","Type":"ContainerStarted","Data":"c5f57e1327fd1093aa74b9628a600ff967a9ccafdf11dd50df96be432ef665ba"}
Oct 30 21:18:53.645478 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:53.645108    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:18:53.683016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:53.682953    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56" podStartSLOduration=2.4051136 podStartE2EDuration="2.682936137s" totalImagesPullingTime="277.822537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:53.681196016 +0000 UTC m=+533.810447467" watchObservedRunningTime="2026-10-30 21:18:53.682936137 +0000 UTC m=+533.812187595"
Oct 30 21:18:55.608420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:55.608382    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-6l9b4"
Oct 30 21:18:55.608919 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:18:55.608457    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9x9cn"
Oct 30 21:19:04.652388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:04.652348    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-845f9f9b5f-ggr56"
Oct 30 21:19:04.714318 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:04.714283    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jwhvc"]
Oct 30 21:19:04.714795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:04.714511    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc" podUID="780269a2-a9c7-4377-9b8a-be67df7871ca" containerName="manager" containerID="cri-o://1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2" gracePeriod=30
Oct 30 21:19:04.960915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:04.960888    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:19:05.033577 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.033546    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780269a2-a9c7-4377-9b8a-be67df7871ca-kube-api-access-6ndbk\" (UniqueName: \"kubernetes.io/projected/780269a2-a9c7-4377-9b8a-be67df7871ca-kube-api-access-6ndbk\") pod \"780269a2-a9c7-4377-9b8a-be67df7871ca\" (UID: \"780269a2-a9c7-4377-9b8a-be67df7871ca\") "
Oct 30 21:19:05.033738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.033591    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/780269a2-a9c7-4377-9b8a-be67df7871ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/780269a2-a9c7-4377-9b8a-be67df7871ca-metrics-certs\") pod \"780269a2-a9c7-4377-9b8a-be67df7871ca\" (UID: \"780269a2-a9c7-4377-9b8a-be67df7871ca\") "
Oct 30 21:19:05.035698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.035675    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780269a2-a9c7-4377-9b8a-be67df7871ca-metrics-certs" pod "780269a2-a9c7-4377-9b8a-be67df7871ca" (UID: "780269a2-a9c7-4377-9b8a-be67df7871ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:05.035811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.035797    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780269a2-a9c7-4377-9b8a-be67df7871ca-kube-api-access-6ndbk" pod "780269a2-a9c7-4377-9b8a-be67df7871ca" (UID: "780269a2-a9c7-4377-9b8a-be67df7871ca"). InnerVolumeSpecName "kube-api-access-6ndbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:05.134435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.134397    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndbk\" (UniqueName: \"kubernetes.io/projected/780269a2-a9c7-4377-9b8a-be67df7871ca-kube-api-access-6ndbk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:19:05.134435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.134428    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780269a2-a9c7-4377-9b8a-be67df7871ca-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:19:05.704651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.704609    5886 generic.go:356] "Generic (PLEG): container finished" podID="780269a2-a9c7-4377-9b8a-be67df7871ca" containerID="1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2" exitCode=0
Oct 30 21:19:05.705051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.704667    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc"
Oct 30 21:19:05.705051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.704680    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc" event={"ID":"780269a2-a9c7-4377-9b8a-be67df7871ca","Type":"ContainerDied","Data":"1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2"}
Oct 30 21:19:05.705051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.704757    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-jwhvc" event={"ID":"780269a2-a9c7-4377-9b8a-be67df7871ca","Type":"ContainerDied","Data":"c2654f54ac3fdb4274a5326226c9c469d98977f84add23cebdc3eeef147d9df4"}
Oct 30 21:19:05.705051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.704784    5886 scope.go:122] "RemoveContainer" containerID="1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2"
Oct 30 21:19:05.714323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.714305    5886 scope.go:122] "RemoveContainer" containerID="1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2"
Oct 30 21:19:05.714594 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:19:05.714576    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2\": container with ID starting with 1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2 not found: ID does not exist" containerID="1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2"
Oct 30 21:19:05.714671 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.714601    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2"} err="failed to get container status \"1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2\": rpc error: code = NotFound desc = could not find container \"1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2\": container with ID starting with 1bef42e8aed8030babd3121d3a31d74235e28298ab5ed93cad16388f27b00fb2 not found: ID does not exist"
Oct 30 21:19:05.751607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.751575    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jwhvc"]
Oct 30 21:19:05.762524 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.762501    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-jwhvc"]
Oct 30 21:19:05.986192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:05.986103    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780269a2-a9c7-4377-9b8a-be67df7871ca" path="/var/lib/kubelet/pods/780269a2-a9c7-4377-9b8a-be67df7871ca/volumes"
Oct 30 21:19:11.027452 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:19:11.027363    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:19:12.133718 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.133679    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"]
Oct 30 21:19:12.134254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.134188    5886 cpu_manager.go:414] "containerMap: removing container" podUID="780269a2-a9c7-4377-9b8a-be67df7871ca" containerName="manager"
Oct 30 21:19:12.134254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.134207    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780269a2-a9c7-4377-9b8a-be67df7871ca" containerName="manager"
Oct 30 21:19:12.134381 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.134320    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="780269a2-a9c7-4377-9b8a-be67df7871ca" containerName="manager"
Oct 30 21:19:12.139028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.139002    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.156361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.156336    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"]
Oct 30 21:19:12.165840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.165811    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:19:12.165840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.165818    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:19:12.166086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.165863    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vj8l8\""
Oct 30 21:19:12.166086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.165999    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 21:19:12.199818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.199788    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.199985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.199831    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.199985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.199854    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.199985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.199914    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6fv\" (UniqueName: \"kubernetes.io/projected/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-kube-api-access-xg6fv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.199985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.199974    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.200131 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.200042    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301251    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6fv\" (UniqueName: \"kubernetes.io/projected/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-kube-api-access-xg6fv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301307    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301336    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301370    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301394    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301417    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301866    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301908    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.301970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.301907    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.303668 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.303648    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.303897 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.303881    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.319009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.318983    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6fv\" (UniqueName: \"kubernetes.io/projected/6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3-kube-api-access-xg6fv\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx\" (UID: \"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.460106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.460063    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:12.759476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:12.759390    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"]
Oct 30 21:19:12.764625 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:19:12.764597    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2f4c05_3262_4b06_adb2_d1ba47b8e1d3.slice/crio-9702913d8465f4da72c8efc1f5535625f649673b25ca1131fccccc2e4575f698 WatchSource:0}: Error finding container 9702913d8465f4da72c8efc1f5535625f649673b25ca1131fccccc2e4575f698: Status 404 returned error can't find the container with id 9702913d8465f4da72c8efc1f5535625f649673b25ca1131fccccc2e4575f698
Oct 30 21:19:13.749040 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:13.749000    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx" event={"ID":"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3","Type":"ContainerStarted","Data":"9702913d8465f4da72c8efc1f5535625f649673b25ca1131fccccc2e4575f698"}
Oct 30 21:19:14.755600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:14.755564    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx" event={"ID":"6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3","Type":"ContainerStarted","Data":"73c419dd137d5d088a332b0ca9debe33f3e9ab9ed1cb9a6e1d9e5a15250e1fe7"}
Oct 30 21:19:14.795877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:14.795829    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx" podStartSLOduration=1.502952536 podStartE2EDuration="2.795817277s" totalImagesPullingTime="1.292864741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:14.792216764 +0000 UTC m=+554.921468222" watchObservedRunningTime="2026-10-30 21:19:14.795817277 +0000 UTC m=+554.925068734"
Oct 30 21:19:15.460644 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:15.460597    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:15.938557 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:15.938525    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"]
Oct 30 21:19:15.944502 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:15.944479    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:15.961677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:15.961098    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"]
Oct 30 21:19:15.970870 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:15.970836    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 21:19:16.037904 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.037868    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.037904 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.037910    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79460cf-0d6b-41f0-9080-22bf5bd7a312-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.038104 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.037991    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lhx\" (UniqueName: \"kubernetes.io/projected/c79460cf-0d6b-41f0-9080-22bf5bd7a312-kube-api-access-d5lhx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.038104 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.038046    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.038104 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.038089    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.038104 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.038104    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.138741 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.138701    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79460cf-0d6b-41f0-9080-22bf5bd7a312-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.138741 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.138742    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lhx\" (UniqueName: \"kubernetes.io/projected/c79460cf-0d6b-41f0-9080-22bf5bd7a312-kube-api-access-d5lhx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.139016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.138768    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.139016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.138813    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.139016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.138829    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.139016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.138866    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.139352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.139318    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.139352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.139338    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.139536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.139384    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.141182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.141132    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79460cf-0d6b-41f0-9080-22bf5bd7a312-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.141331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.141310    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79460cf-0d6b-41f0-9080-22bf5bd7a312-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.157195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.157141    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lhx\" (UniqueName: \"kubernetes.io/projected/c79460cf-0d6b-41f0-9080-22bf5bd7a312-kube-api-access-d5lhx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gp7xk\" (UID: \"c79460cf-0d6b-41f0-9080-22bf5bd7a312\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.266993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.266902    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:16.561025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.560931    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"]
Oct 30 21:19:16.565479 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:19:16.565447    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79460cf_0d6b_41f0_9080_22bf5bd7a312.slice/crio-33a6cc0e67c23fe6f9baddb8a92aca1260c6652730b1aec74dbf57ae6184823d WatchSource:0}: Error finding container 33a6cc0e67c23fe6f9baddb8a92aca1260c6652730b1aec74dbf57ae6184823d: Status 404 returned error can't find the container with id 33a6cc0e67c23fe6f9baddb8a92aca1260c6652730b1aec74dbf57ae6184823d
Oct 30 21:19:16.766812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:16.766770    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk" event={"ID":"c79460cf-0d6b-41f0-9080-22bf5bd7a312","Type":"ContainerStarted","Data":"33a6cc0e67c23fe6f9baddb8a92aca1260c6652730b1aec74dbf57ae6184823d"}
Oct 30 21:19:17.774622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:17.774590    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk" event={"ID":"c79460cf-0d6b-41f0-9080-22bf5bd7a312","Type":"ContainerStarted","Data":"bac8b4add65fdc28b7123b3eabc373284b1a57d09d355ba175ab528a86ff1ed3"}
Oct 30 21:19:17.813053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:17.813004    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk" podStartSLOduration=2.600359851 podStartE2EDuration="2.812989565s" totalImagesPullingTime="212.629714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:17.811377428 +0000 UTC m=+557.940628897" watchObservedRunningTime="2026-10-30 21:19:17.812989565 +0000 UTC m=+557.942241065"
Oct 30 21:19:18.267657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.267621    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:18.536207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.536105    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"]
Oct 30 21:19:18.540509 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.540485    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.557044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.557019    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 21:19:18.569775 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.569751    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"]
Oct 30 21:19:18.663417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.663379    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.663417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.663419    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.663650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.663441    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.663650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.663458    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.663650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.663595    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5x2\" (UniqueName: \"kubernetes.io/projected/af485e41-5b2c-4b83-9bce-972e34da300d-kube-api-access-zb5x2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.663650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.663620    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af485e41-5b2c-4b83-9bce-972e34da300d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.764788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.764754    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5x2\" (UniqueName: \"kubernetes.io/projected/af485e41-5b2c-4b83-9bce-972e34da300d-kube-api-access-zb5x2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.764974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.764793    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af485e41-5b2c-4b83-9bce-972e34da300d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.764974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.764850    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.764974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.764866    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.764974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.764897    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.764974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.764920    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.765324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.765303    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.765414 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.765396    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.765465 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.765397    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.767368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.767345    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af485e41-5b2c-4b83-9bce-972e34da300d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.767644 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.767621    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af485e41-5b2c-4b83-9bce-972e34da300d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.782888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.782862    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5x2\" (UniqueName: \"kubernetes.io/projected/af485e41-5b2c-4b83-9bce-972e34da300d-kube-api-access-zb5x2\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-d44dq\" (UID: \"af485e41-5b2c-4b83-9bce-972e34da300d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:18.855043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:18.854956    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:19.151772 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:19.151744    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"]
Oct 30 21:19:19.156902 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:19:19.156877    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf485e41_5b2c_4b83_9bce_972e34da300d.slice/crio-71b30a56a68d6fb10e017529466fd307369b3ca14868dba025863c7a4457291b WatchSource:0}: Error finding container 71b30a56a68d6fb10e017529466fd307369b3ca14868dba025863c7a4457291b: Status 404 returned error can't find the container with id 71b30a56a68d6fb10e017529466fd307369b3ca14868dba025863c7a4457291b
Oct 30 21:19:19.788372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:19.788279    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq" event={"ID":"af485e41-5b2c-4b83-9bce-972e34da300d","Type":"ContainerStarted","Data":"20638088703b24bd41f409695f8b084e80d1f1d6aba9a9e7d1689df43261f44d"}
Oct 30 21:19:19.788372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:19.788332    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq" event={"ID":"af485e41-5b2c-4b83-9bce-972e34da300d","Type":"ContainerStarted","Data":"71b30a56a68d6fb10e017529466fd307369b3ca14868dba025863c7a4457291b"}
Oct 30 21:19:19.825543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:19.825489    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq" podStartSLOduration=1.6148539450000001 podStartE2EDuration="1.825474936s" totalImagesPullingTime="210.620991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:19.825049413 +0000 UTC m=+559.954300871" watchObservedRunningTime="2026-10-30 21:19:19.825474936 +0000 UTC m=+559.954726392"
Oct 30 21:19:19.855797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:19.855761    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:22.461004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:22.460968    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:22.473542 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:22.473514    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:22.818633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:22.818556    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx"
Oct 30 21:19:26.267238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:26.267195    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:26.287470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:26.287442    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:26.843222 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:26.843114    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gp7xk"
Oct 30 21:19:28.274679 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.274639    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-79kvw"]
Oct 30 21:19:28.279269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.279242    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.293308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.293280    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 21:19:28.306054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.306028    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-79kvw"]
Oct 30 21:19:28.349925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.349894    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxshd\" (UniqueName: \"kubernetes.io/projected/0cebd872-e218-4bd4-a990-0c22fc0afb6e-kube-api-access-bxshd\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.350102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.349952    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.350102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.350041    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.350102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.350067    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-home\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.350308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.350222    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.350308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.350258    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cebd872-e218-4bd4-a990-0c22fc0afb6e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.451694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.451654    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.451856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.451702    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cebd872-e218-4bd4-a990-0c22fc0afb6e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.451856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.451732    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxshd\" (UniqueName: \"kubernetes.io/projected/0cebd872-e218-4bd4-a990-0c22fc0afb6e-kube-api-access-bxshd\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.451856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.451801    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.451856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.451842    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.452062 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.451867    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-home\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.452134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.452111    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.452243 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.452224    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.452304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.452283    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-home\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.454068 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.454049    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cebd872-e218-4bd4-a990-0c22fc0afb6e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.454472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.454450    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cebd872-e218-4bd4-a990-0c22fc0afb6e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.469577 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.469518    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxshd\" (UniqueName: \"kubernetes.io/projected/0cebd872-e218-4bd4-a990-0c22fc0afb6e-kube-api-access-bxshd\") pod \"echo-simulator-kserve-64b6f8f96c-79kvw\" (UID: \"0cebd872-e218-4bd4-a990-0c22fc0afb6e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.593897 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.593806    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:28.855567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.855486    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:28.868219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.868196    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:28.889907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:28.889878    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-79kvw"]
Oct 30 21:19:28.894919 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:19:28.894894    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cebd872_e218_4bd4_a990_0c22fc0afb6e.slice/crio-ad09d78608a65f0109aad821d0422e4f4b1e694ebfd14564af56cb6c5b9fa5b3 WatchSource:0}: Error finding container ad09d78608a65f0109aad821d0422e4f4b1e694ebfd14564af56cb6c5b9fa5b3: Status 404 returned error can't find the container with id ad09d78608a65f0109aad821d0422e4f4b1e694ebfd14564af56cb6c5b9fa5b3
Oct 30 21:19:29.853235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:29.853191    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw" event={"ID":"0cebd872-e218-4bd4-a990-0c22fc0afb6e","Type":"ContainerStarted","Data":"ad09d78608a65f0109aad821d0422e4f4b1e694ebfd14564af56cb6c5b9fa5b3"}
Oct 30 21:19:29.867474 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:29.867443    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-d44dq"
Oct 30 21:19:30.860609 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:30.860514    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw" event={"ID":"0cebd872-e218-4bd4-a990-0c22fc0afb6e","Type":"ContainerStarted","Data":"31e49daaecabcee25f38b04d7dbc4a40d2084d13b4becc967483e00c833ef65d"}
Oct 30 21:19:30.898474 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:30.898423    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw" podStartSLOduration=1.23483861 podStartE2EDuration="2.898410261s" totalImagesPullingTime="1.663571651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:30.896978625 +0000 UTC m=+571.026230082" watchObservedRunningTime="2026-10-30 21:19:30.898410261 +0000 UTC m=+571.027661745"
Oct 30 21:19:31.374653 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.374600    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"]
Oct 30 21:19:31.379077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.379049    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.398973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.398939    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"]
Oct 30 21:19:31.409771 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.409666    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 21:19:31.480659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.480613    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24915d9d-898e-4455-8c32-a4ff2bac9251-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.480865 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.480668    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.480865 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.480735    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.480995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.480885    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.480995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.480931    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.481100 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.481015    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5dv\" (UniqueName: \"kubernetes.io/projected/24915d9d-898e-4455-8c32-a4ff2bac9251-kube-api-access-kh5dv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.581702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.581658    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.581995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.581970    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.582143 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.582126    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5dv\" (UniqueName: \"kubernetes.io/projected/24915d9d-898e-4455-8c32-a4ff2bac9251-kube-api-access-kh5dv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.582346 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.582326    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24915d9d-898e-4455-8c32-a4ff2bac9251-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.582530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.582512    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.582638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.582328    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.582715 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.582625    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.583244 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.583223    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.583466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.583443    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.588576 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.588389    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24915d9d-898e-4455-8c32-a4ff2bac9251-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.588785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.588765    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24915d9d-898e-4455-8c32-a4ff2bac9251-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.594167 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.594071    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:31.605501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.605474    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5dv\" (UniqueName: \"kubernetes.io/projected/24915d9d-898e-4455-8c32-a4ff2bac9251-kube-api-access-kh5dv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c\" (UID: \"24915d9d-898e-4455-8c32-a4ff2bac9251\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:31.696792 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:31.696748    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:32.009314 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:32.007553    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"]
Oct 30 21:19:32.013330 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:19:32.013256    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24915d9d_898e_4455_8c32_a4ff2bac9251.slice/crio-622a588ac864834462ce5b51f49256b3c821c11753e89714aa64fedd87f51ff3 WatchSource:0}: Error finding container 622a588ac864834462ce5b51f49256b3c821c11753e89714aa64fedd87f51ff3: Status 404 returned error can't find the container with id 622a588ac864834462ce5b51f49256b3c821c11753e89714aa64fedd87f51ff3
Oct 30 21:19:32.876359 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:32.876322    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" event={"ID":"24915d9d-898e-4455-8c32-a4ff2bac9251","Type":"ContainerStarted","Data":"971ee7cd0a80f740dba3f1405b911c667c44f782875c22a674de5c31d62b194e"}
Oct 30 21:19:32.876359 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:32.876366    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" event={"ID":"24915d9d-898e-4455-8c32-a4ff2bac9251","Type":"ContainerStarted","Data":"622a588ac864834462ce5b51f49256b3c821c11753e89714aa64fedd87f51ff3"}
Oct 30 21:19:32.922082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:32.922035    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podStartSLOduration=1.692405553 podStartE2EDuration="1.922021569s" totalImagesPullingTime="229.616016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:32.92142456 +0000 UTC m=+573.050676017" watchObservedRunningTime="2026-10-30 21:19:32.922021569 +0000 UTC m=+573.051273025"
Oct 30 21:19:33.697813 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:33.697768    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:38.594707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:38.594668    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:38.607488 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:38.607463    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:38.919378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:38.919350    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-79kvw"
Oct 30 21:19:39.297707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.297607    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"]
Oct 30 21:19:39.301768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.301746    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.332960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.332928    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"]
Oct 30 21:19:39.463292 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463252    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.463292 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463289    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.463522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463308    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.463522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463395    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.463522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463453    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.463522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463487    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.463522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463510    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.463748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463615    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.463748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.463640    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2s6\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-kube-api-access-9g2s6\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.564999    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565043    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2s6\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-kube-api-access-9g2s6\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565076    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565193    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565224    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565257    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565291    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565346    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565622    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565644    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565614    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565670    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.565899 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565861    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.566004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.565985    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.567636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.567612    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.567798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.567777    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.584653 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.584624    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.584954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.584931    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2s6\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-kube-api-access-9g2s6\") pod \"maas-default-gateway-openshift-default-77f556d647-6qx95\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.616741 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.616718    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:39.932059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.932022    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"]
Oct 30 21:19:39.938984 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:19:39.938957    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370db4fb_48b4_4d14_9200_b2d84e12c9f0.slice/crio-49734d8aa377e8ab0559c14ccca1352890dff172d1284fb1493ac6a35e36c625 WatchSource:0}: Error finding container 49734d8aa377e8ab0559c14ccca1352890dff172d1284fb1493ac6a35e36c625: Status 404 returned error can't find the container with id 49734d8aa377e8ab0559c14ccca1352890dff172d1284fb1493ac6a35e36c625
Oct 30 21:19:39.941272 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.941243    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:39.941360 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.941307    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:39.941360 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:39.941336    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:40.919520 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:40.919484    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" event={"ID":"370db4fb-48b4-4d14-9200-b2d84e12c9f0","Type":"ContainerStarted","Data":"5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e"}
Oct 30 21:19:40.919997 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:40.919550    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" event={"ID":"370db4fb-48b4-4d14-9200-b2d84e12c9f0","Type":"ContainerStarted","Data":"49734d8aa377e8ab0559c14ccca1352890dff172d1284fb1493ac6a35e36c625"}
Oct 30 21:19:40.960447 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:40.960382    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podStartSLOduration=1.96035194 podStartE2EDuration="1.96035194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:40.958550942 +0000 UTC m=+581.087802391" watchObservedRunningTime="2026-10-30 21:19:40.96035194 +0000 UTC m=+581.089603388"
Oct 30 21:19:41.027360 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:19:41.027321    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:19:41.617202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:41.617139    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:41.618773 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:41.618747    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6qx95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:19:41.618891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:41.618803    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:19:41.696947 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:41.696917    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:41.710232 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:41.710205    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:41.935618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:41.935594    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:19:42.617569 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:42.617526    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6qx95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:19:42.617787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:42.617593    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:19:43.617543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:43.617504    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6qx95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:19:43.617928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:43.617560    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:19:44.617560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:44.617524    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6qx95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:19:44.617956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:44.617587    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:19:45.617106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:45.617075    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6qx95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:19:45.617297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:45.617127    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:19:46.617704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:46.617664    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6qx95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:19:46.618127 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:46.617734    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:19:48.294414 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:48.294379    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:48.295169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:48.295127    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:48.296082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:48.296052    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:19:54.359617 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.359536    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"]
Oct 30 21:19:54.375925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.375888    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.390239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.390179    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"]
Oct 30 21:19:54.415043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415011    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.415198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415054    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822k4\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-kube-api-access-822k4\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.415198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415081    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.415280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415237    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.415280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415259    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c342234-2531-49aa-b054-e7be98b2b36d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.415280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415278    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.415447 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415419    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c342234-2531-49aa-b054-e7be98b2b36d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.415532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415515    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.415603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.415588    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.516955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.516916    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822k4\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-kube-api-access-822k4\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.516955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.516959    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517057    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517081    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c342234-2531-49aa-b054-e7be98b2b36d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517109    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517147    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c342234-2531-49aa-b054-e7be98b2b36d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517221    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517493 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517306    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517493 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517350    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517538    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517616    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517803    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517933 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517907    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c342234-2531-49aa-b054-e7be98b2b36d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.517993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.517936    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.519873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.519854    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.519873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.519862    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c342234-2531-49aa-b054-e7be98b2b36d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.534812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.534782    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.534976 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.534955    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-822k4\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-kube-api-access-822k4\") pod \"maas-default-gateway-openshift-default-77f556d647-2bqcx\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:54.692837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:54.692800    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:55.010473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:55.010375    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"]
Oct 30 21:19:55.019582 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:19:55.019554    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c342234_2531_49aa_b054_e7be98b2b36d.slice/crio-c6aaa3312d3b50406ba792480e91d1f7e08502c0ee4791c1184a5def6be709ea WatchSource:0}: Error finding container c6aaa3312d3b50406ba792480e91d1f7e08502c0ee4791c1184a5def6be709ea: Status 404 returned error can't find the container with id c6aaa3312d3b50406ba792480e91d1f7e08502c0ee4791c1184a5def6be709ea
Oct 30 21:19:55.021534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:55.021494    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:55.021657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:55.021582    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:55.021657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:55.021627    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:19:56.008078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:56.008042    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" event={"ID":"9c342234-2531-49aa-b054-e7be98b2b36d","Type":"ContainerStarted","Data":"cb991ccbc1242c6e6948e2e4f96fa17f56a67cbcd7a1ac2ad088b6627da65e92"}
Oct 30 21:19:56.008504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:56.008098    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" event={"ID":"9c342234-2531-49aa-b054-e7be98b2b36d","Type":"ContainerStarted","Data":"c6aaa3312d3b50406ba792480e91d1f7e08502c0ee4791c1184a5def6be709ea"}
Oct 30 21:19:56.060385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:56.060332    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podStartSLOduration=2.060319089 podStartE2EDuration="2.060319089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:56.058481126 +0000 UTC m=+596.187732583" watchObservedRunningTime="2026-10-30 21:19:56.060319089 +0000 UTC m=+596.189570546"
Oct 30 21:19:56.693709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:56.693673    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:19:56.695619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:56.695590    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:19:56.695775 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:56.695640    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:19:57.693255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:57.693212    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:19:57.693657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:57.693271    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:19:58.693568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:58.693529    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:19:58.694028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:58.693600    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:19:59.693245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:59.693207    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:19:59.693459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:19:59.693281    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:20:00.693442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:00.693391    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:20:00.693951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:00.693459    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:20:01.693882 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:01.693838    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:20:01.694297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:01.693908    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:20:03.204600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:03.204566    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:20:03.205088 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:03.205008    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:20:03.206141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:03.206119    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:20:11.027580 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:11.027533    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:20:15.032815 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.032774    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"]
Oct 30 21:20:15.036533 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.036508    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.053724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.053684    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bb245e02-openshift-default-dockercfg-dhdx5\""
Oct 30 21:20:15.066199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.066141    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"]
Oct 30 21:20:15.130032 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.129998    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-certs\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.130255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.130043    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-credential-socket\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.130255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.130233    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-data\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.130455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.130281    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwfhj\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-kube-api-access-gwfhj\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.130455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.130397    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4b7e15-3adb-410b-b1da-a089490823af-istio-podinfo\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.130455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.130427    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4b7e15-3adb-410b-b1da-a089490823af-istiod-ca-cert\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.130605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.130487    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-istio-token\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.130605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.130535    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-envoy\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.130705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.130623    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-socket\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231389    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-credential-socket\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231470    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-data\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231646    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwfhj\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-kube-api-access-gwfhj\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231700    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4b7e15-3adb-410b-b1da-a089490823af-istio-podinfo\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231792 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231731    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4b7e15-3adb-410b-b1da-a089490823af-istiod-ca-cert\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231792 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231761    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-istio-token\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231792 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231779    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-data\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231822    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-envoy\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231857    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-socket\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231859    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-credential-socket\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.231938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.231878    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-certs\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.232129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.232066    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-certs\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.232338 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.232312    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-socket\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.232800 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.232773    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4b7e15-3adb-410b-b1da-a089490823af-istiod-ca-cert\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.234908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.234882    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4b7e15-3adb-410b-b1da-a089490823af-istio-podinfo\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.235063 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.235047    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-envoy\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.251062 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.251026    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-istio-token\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.251725 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.251704    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwfhj\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-kube-api-access-gwfhj\") pod \"e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") " pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.352419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.352331    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:15.552781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.552670    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"]
Oct 30 21:20:15.558478 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.558450    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.576536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.576509    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bdcfca55-gw-openshift-default-dockercfg-h7skc\""
Oct 30 21:20:15.602975 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.602878    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"]
Oct 30 21:20:15.637471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637440    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-certs\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.637635 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637476    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istiod-ca-cert\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.637635 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637501    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-socket\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.637754 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637650    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-credential-socket\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.637754 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637692    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-podinfo\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.637847 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637752    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-data\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.637847 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637784    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrcjq\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-kube-api-access-hrcjq\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.637948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637890    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-envoy\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.637948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.637921    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-token\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.703293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.703257    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"]
Oct 30 21:20:15.708866 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:15.708830    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4b7e15_3adb_410b_b1da_a089490823af.slice/crio-352e7f6fb2701a1eaf861c2804208e11786b6023748a846b824993c234d2dcdd WatchSource:0}: Error finding container 352e7f6fb2701a1eaf861c2804208e11786b6023748a846b824993c234d2dcdd: Status 404 returned error can't find the container with id 352e7f6fb2701a1eaf861c2804208e11786b6023748a846b824993c234d2dcdd
Oct 30 21:20:15.711913 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.711881    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:15.711993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.711945    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:15.711993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.711972    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:15.738579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738550    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-certs\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.738707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738592    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istiod-ca-cert\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.738707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738619    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-socket\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.738707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738655    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-credential-socket\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.738707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738683    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-podinfo\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.738951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738714    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-data\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.738951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738777    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrcjq\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-kube-api-access-hrcjq\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.738951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738862    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-envoy\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.738951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.738889    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-token\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.739141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.739010    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-certs\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.739141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.739101    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-data\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.739430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.739405    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istiod-ca-cert\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.739503 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.739440    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-socket\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.739503 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.739412    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-credential-socket\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.741029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.741008    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-envoy\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.741389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.741367    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-podinfo\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.759349 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.759321    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-token\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.759448 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.759381    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrcjq\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-kube-api-access-hrcjq\") pod \"e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") " pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:15.876775 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:15.876685    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:16.117999 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.117950    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" event={"ID":"7c4b7e15-3adb-410b-b1da-a089490823af","Type":"ContainerStarted","Data":"10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63"}
Oct 30 21:20:16.118500 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.118021    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" event={"ID":"7c4b7e15-3adb-410b-b1da-a089490823af","Type":"ContainerStarted","Data":"352e7f6fb2701a1eaf861c2804208e11786b6023748a846b824993c234d2dcdd"}
Oct 30 21:20:16.159188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.159102    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podStartSLOduration=1.159085075 podStartE2EDuration="1.159085075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:16.15685418 +0000 UTC m=+616.286105641" watchObservedRunningTime="2026-10-30 21:20:16.159085075 +0000 UTC m=+616.288336529"
Oct 30 21:20:16.206103 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.206063    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"]
Oct 30 21:20:16.211165 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:16.211117    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0673bbbf_2ea9_47f4_918d_3b17ad723cdd.slice/crio-2a71732b8ab6d5ba15691203aaa4c9a3c19c9db85949828e5ea088b25d696a57 WatchSource:0}: Error finding container 2a71732b8ab6d5ba15691203aaa4c9a3c19c9db85949828e5ea088b25d696a57: Status 404 returned error can't find the container with id 2a71732b8ab6d5ba15691203aaa4c9a3c19c9db85949828e5ea088b25d696a57
Oct 30 21:20:16.213982 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.213938    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:16.214072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.214028    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:16.214115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.214068    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:16.352922 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.352886    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:16.358473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.358447    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:16.847340 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.847300    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"]
Oct 30 21:20:16.852003 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.851973    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.869867 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.869844    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b72f7bd8-openshift-default-dockercfg-fw85q\""
Oct 30 21:20:16.883650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.883624    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"]
Oct 30 21:20:16.951421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951392    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-podinfo\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.951421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951421    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-token\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.951421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951441    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istiod-ca-cert\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.951730 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951461    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-socket\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.951730 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951624    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-certs\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.951730 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951720    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-data\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.951944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951766    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-credential-socket\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.951944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951896    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-envoy\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:16.951944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:16.951922    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-kube-api-access-xglkb\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053272    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-data\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053316    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-credential-socket\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053377    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-envoy\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053394    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-kube-api-access-xglkb\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053443    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-podinfo\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053468    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-token\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053494    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istiod-ca-cert\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053527    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-socket\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053591    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-certs\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053901 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053777    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-credential-socket\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.053963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.053938    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-certs\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.054064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.054042    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-socket\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.054125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.054088    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-data\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.054277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.054258    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istiod-ca-cert\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.055617 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.055599    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-envoy\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.055842 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.055816    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-podinfo\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.076103 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.075918    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-token\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.078636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.078581    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-kube-api-access-xglkb\") pod \"e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") " pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.126581 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.126456    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" event={"ID":"0673bbbf-2ea9-47f4-918d-3b17ad723cdd","Type":"ContainerStarted","Data":"313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd"}
Oct 30 21:20:17.126581 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.126529    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" event={"ID":"0673bbbf-2ea9-47f4-918d-3b17ad723cdd","Type":"ContainerStarted","Data":"2a71732b8ab6d5ba15691203aaa4c9a3c19c9db85949828e5ea088b25d696a57"}
Oct 30 21:20:17.127179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.126818    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:17.127868 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.127846    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:17.167412 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.167367    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" podStartSLOduration=2.167354923 podStartE2EDuration="2.167354923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:17.166033166 +0000 UTC m=+617.295284623" watchObservedRunningTime="2026-10-30 21:20:17.167354923 +0000 UTC m=+617.296606380"
Oct 30 21:20:17.183017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.182989    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:17.542515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.542482    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"]
Oct 30 21:20:17.546660 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:17.546633    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a50620_2814_4dc4_a8c9_493a6d4ea03a.slice/crio-0f47c89216e40dd59bd38b722005940204e3669c2aa1d7fae6393c00809ceb5f WatchSource:0}: Error finding container 0f47c89216e40dd59bd38b722005940204e3669c2aa1d7fae6393c00809ceb5f: Status 404 returned error can't find the container with id 0f47c89216e40dd59bd38b722005940204e3669c2aa1d7fae6393c00809ceb5f
Oct 30 21:20:17.549457 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.549427    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:17.549521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.549513    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:17.549558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.549542    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:17.878712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.878616    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:17.884286 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:17.884260    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:18.133224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:18.133094    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" event={"ID":"82a50620-2814-4dc4-a8c9-493a6d4ea03a","Type":"ContainerStarted","Data":"4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60"}
Oct 30 21:20:18.133224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:18.133150    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" event={"ID":"82a50620-2814-4dc4-a8c9-493a6d4ea03a","Type":"ContainerStarted","Data":"0f47c89216e40dd59bd38b722005940204e3669c2aa1d7fae6393c00809ceb5f"}
Oct 30 21:20:18.133744 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:18.133566    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:18.134308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:18.134291    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:18.173432 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:18.173383    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" podStartSLOduration=2.1733708 podStartE2EDuration="2.1733708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:18.172029819 +0000 UTC m=+618.301281275" watchObservedRunningTime="2026-10-30 21:20:18.1733708 +0000 UTC m=+618.302622315"
Oct 30 21:20:18.183258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:18.183228    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:18.189643 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:18.189621    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:19.138362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.138324    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:19.139372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.139349    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:19.737864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.737828    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"]
Oct 30 21:20:19.741827 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.741804    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:19.757974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.757950    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bb245e02-dockercfg-zndwg\""
Oct 30 21:20:19.760414 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.760004    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"]
Oct 30 21:20:19.787565 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.787535    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3485681-3d15-4267-911b-692beabee729-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n\" (UID: \"f3485681-3d15-4267-911b-692beabee729\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:19.787712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.787632    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67bb\" (UniqueName: \"kubernetes.io/projected/f3485681-3d15-4267-911b-692beabee729-kube-api-access-f67bb\") pod \"payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n\" (UID: \"f3485681-3d15-4267-911b-692beabee729\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:19.796420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.796389    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bb245e02\""
Oct 30 21:20:19.888640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.888609    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3485681-3d15-4267-911b-692beabee729-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n\" (UID: \"f3485681-3d15-4267-911b-692beabee729\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:19.888807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.888654    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67bb\" (UniqueName: \"kubernetes.io/projected/f3485681-3d15-4267-911b-692beabee729-kube-api-access-f67bb\") pod \"payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n\" (UID: \"f3485681-3d15-4267-911b-692beabee729\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:19.889376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.889353    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3485681-3d15-4267-911b-692beabee729-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n\" (UID: \"f3485681-3d15-4267-911b-692beabee729\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:19.906840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:19.906809    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67bb\" (UniqueName: \"kubernetes.io/projected/f3485681-3d15-4267-911b-692beabee729-kube-api-access-f67bb\") pod \"payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n\" (UID: \"f3485681-3d15-4267-911b-692beabee729\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:20.055648 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:20.055565    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:20.360238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:20.360129    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"]
Oct 30 21:20:20.364997 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:20.364963    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3485681_3d15_4267_911b_692beabee729.slice/crio-088837a0f7cb60dbe67c3529cde0f441bce432f8bae572e07d58a58861e77bfc WatchSource:0}: Error finding container 088837a0f7cb60dbe67c3529cde0f441bce432f8bae572e07d58a58861e77bfc: Status 404 returned error can't find the container with id 088837a0f7cb60dbe67c3529cde0f441bce432f8bae572e07d58a58861e77bfc
Oct 30 21:20:21.154801 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.154764    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n" event={"ID":"f3485681-3d15-4267-911b-692beabee729","Type":"ContainerStarted","Data":"ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c"}
Oct 30 21:20:21.154996 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.154825    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n" event={"ID":"f3485681-3d15-4267-911b-692beabee729","Type":"ContainerStarted","Data":"088837a0f7cb60dbe67c3529cde0f441bce432f8bae572e07d58a58861e77bfc"}
Oct 30 21:20:21.154996 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.154864    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:21.206955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.206896    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n" podStartSLOduration=2.206881744 podStartE2EDuration="2.206881744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:21.204179028 +0000 UTC m=+621.333430482" watchObservedRunningTime="2026-10-30 21:20:21.206881744 +0000 UTC m=+621.336133234"
Oct 30 21:20:21.320362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.320317    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"]
Oct 30 21:20:21.324234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.324201    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.343345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.343316    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"]
Oct 30 21:20:21.349575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.349541    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bdcfca55-serving-cert\""
Oct 30 21:20:21.404873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.404783    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-maas-api-tls\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.404873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.404848    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-metrics-certs\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.405348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.404944    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76961f99-09f5-45df-a9b7-156dcd5008e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.405348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.404976    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gg5d\" (UniqueName: \"kubernetes.io/projected/76961f99-09f5-45df-a9b7-156dcd5008e8-kube-api-access-9gg5d\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.506739 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.506695    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-maas-api-tls\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.506930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.506752    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-metrics-certs\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.506930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.506789    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76961f99-09f5-45df-a9b7-156dcd5008e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.506930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.506810    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gg5d\" (UniqueName: \"kubernetes.io/projected/76961f99-09f5-45df-a9b7-156dcd5008e8-kube-api-access-9gg5d\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.507581 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.507559    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76961f99-09f5-45df-a9b7-156dcd5008e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.509218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.509196    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-maas-api-tls\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.509299 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.509237    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-metrics-certs\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.532464 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.532435    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gg5d\" (UniqueName: \"kubernetes.io/projected/76961f99-09f5-45df-a9b7-156dcd5008e8-kube-api-access-9gg5d\") pod \"maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.637779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.637743    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:21.944537 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.944355    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"]
Oct 30 21:20:21.948626 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:21.948592    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76961f99_09f5_45df_a9b7_156dcd5008e8.slice/crio-04bfb47dc49de999d8af3f3810d0cdc0401a479fe2e9ab10354024717da9c280 WatchSource:0}: Error finding container 04bfb47dc49de999d8af3f3810d0cdc0401a479fe2e9ab10354024717da9c280: Status 404 returned error can't find the container with id 04bfb47dc49de999d8af3f3810d0cdc0401a479fe2e9ab10354024717da9c280
Oct 30 21:20:21.950003 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:21.949982    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:20:22.166279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:22.166239    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq" event={"ID":"76961f99-09f5-45df-a9b7-156dcd5008e8","Type":"ContainerStarted","Data":"04bfb47dc49de999d8af3f3810d0cdc0401a479fe2e9ab10354024717da9c280"}
Oct 30 21:20:23.200646 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.200611    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq" event={"ID":"76961f99-09f5-45df-a9b7-156dcd5008e8","Type":"ContainerStarted","Data":"1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20"}
Oct 30 21:20:23.201181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.200677    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:23.241850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.241795    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq" podStartSLOduration=1.907754089 podStartE2EDuration="2.241777165s" totalImagesPullingTime="334.023076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:23.238306957 +0000 UTC m=+623.367558415" watchObservedRunningTime="2026-10-30 21:20:23.241777165 +0000 UTC m=+623.371028624"
Oct 30 21:20:23.431231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.431198    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"]
Oct 30 21:20:23.434938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.434917    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:23.460428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.460341    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"]
Oct 30 21:20:23.473964 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.473488    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b72f7bd8-dockercfg-xfdbb\""
Oct 30 21:20:23.473964 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.473578    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b72f7bd8\""
Oct 30 21:20:23.534200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.532090    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ea7856-ad34-4014-8e37-6bb81955a2a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz\" (UID: \"55ea7856-ad34-4014-8e37-6bb81955a2a9\") " pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:23.534200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.532202    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd7b9\" (UniqueName: \"kubernetes.io/projected/55ea7856-ad34-4014-8e37-6bb81955a2a9-kube-api-access-fd7b9\") pod \"payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz\" (UID: \"55ea7856-ad34-4014-8e37-6bb81955a2a9\") " pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:23.633548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.633513    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd7b9\" (UniqueName: \"kubernetes.io/projected/55ea7856-ad34-4014-8e37-6bb81955a2a9-kube-api-access-fd7b9\") pod \"payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz\" (UID: \"55ea7856-ad34-4014-8e37-6bb81955a2a9\") " pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:23.633742 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.633634    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ea7856-ad34-4014-8e37-6bb81955a2a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz\" (UID: \"55ea7856-ad34-4014-8e37-6bb81955a2a9\") " pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:23.634423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.634397    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ea7856-ad34-4014-8e37-6bb81955a2a9-plugins-config-volume\") pod \"payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz\" (UID: \"55ea7856-ad34-4014-8e37-6bb81955a2a9\") " pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:23.651896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.651867    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd7b9\" (UniqueName: \"kubernetes.io/projected/55ea7856-ad34-4014-8e37-6bb81955a2a9-kube-api-access-fd7b9\") pod \"payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz\" (UID: \"55ea7856-ad34-4014-8e37-6bb81955a2a9\") " pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:23.747948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:23.747851    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:24.090227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:24.086021    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"]
Oct 30 21:20:24.092841 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:24.092810    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ea7856_ad34_4014_8e37_6bb81955a2a9.slice/crio-f5aa4bed3782d2e9bc2b3fb1dc7fb9b6dd52644c7c36010444a05383b19e3d99 WatchSource:0}: Error finding container f5aa4bed3782d2e9bc2b3fb1dc7fb9b6dd52644c7c36010444a05383b19e3d99: Status 404 returned error can't find the container with id f5aa4bed3782d2e9bc2b3fb1dc7fb9b6dd52644c7c36010444a05383b19e3d99
Oct 30 21:20:24.212178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:24.211238    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz" event={"ID":"55ea7856-ad34-4014-8e37-6bb81955a2a9","Type":"ContainerStarted","Data":"0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f"}
Oct 30 21:20:24.212178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:24.211311    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz" event={"ID":"55ea7856-ad34-4014-8e37-6bb81955a2a9","Type":"ContainerStarted","Data":"f5aa4bed3782d2e9bc2b3fb1dc7fb9b6dd52644c7c36010444a05383b19e3d99"}
Oct 30 21:20:25.216449 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:25.216411    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:25.254065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:25.254015    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz" podStartSLOduration=2.253999051 podStartE2EDuration="2.253999051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:25.252331424 +0000 UTC m=+625.381582882" watchObservedRunningTime="2026-10-30 21:20:25.253999051 +0000 UTC m=+625.383250511"
Oct 30 21:20:28.752633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.752590    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"]
Oct 30 21:20:28.758558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.758530    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.794371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.794340    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"]
Oct 30 21:20:28.828892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.828863    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-maas-api-tls\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.828892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.828895    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-metrics-certs\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.829128 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.828919    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvcg\" (UniqueName: \"kubernetes.io/projected/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-kube-api-access-9rvcg\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.829128 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.828992    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-openshift-service-ca\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.844616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.844588    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-acb99762-serving-cert\""
Oct 30 21:20:28.930311 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.930273    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-maas-api-tls\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.930311 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.930323    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-metrics-certs\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.930572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.930363    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvcg\" (UniqueName: \"kubernetes.io/projected/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-kube-api-access-9rvcg\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.930572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.930500    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-openshift-service-ca\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.931168 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.931123    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-openshift-service-ca\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.932948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.932923    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-metrics-certs\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.935837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.935812    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-maas-api-tls\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:28.949878 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:28.949802    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvcg\" (UniqueName: \"kubernetes.io/projected/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-kube-api-access-9rvcg\") pod \"maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:29.111563 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:29.111517    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:29.229228 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:29.228283    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:29.451933 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:29.451840    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"]
Oct 30 21:20:29.456031 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:29.456000    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb7dfa2_8f6e_4c9e_9ddb_79e6f55848a6.slice/crio-554a06355672422595918aecfe0adc5810b72149942bfa13622cef12c4be39ad WatchSource:0}: Error finding container 554a06355672422595918aecfe0adc5810b72149942bfa13622cef12c4be39ad: Status 404 returned error can't find the container with id 554a06355672422595918aecfe0adc5810b72149942bfa13622cef12c4be39ad
Oct 30 21:20:30.256950 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:30.256912    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc" event={"ID":"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6","Type":"ContainerStarted","Data":"df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1"}
Oct 30 21:20:30.257395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:30.256968    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc" event={"ID":"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6","Type":"ContainerStarted","Data":"554a06355672422595918aecfe0adc5810b72149942bfa13622cef12c4be39ad"}
Oct 30 21:20:30.257395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:30.257094    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:30.303643 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:30.303581    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc" podStartSLOduration=1.974417141 podStartE2EDuration="2.303563215s" totalImagesPullingTime="329.146074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:30.300376018 +0000 UTC m=+630.429627476" watchObservedRunningTime="2026-10-30 21:20:30.303563215 +0000 UTC m=+630.432814672"
Oct 30 21:20:32.188959 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:32.188925    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:33.176643 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:33.176270    5886 patch_prober.go:29] interesting pod/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:20:33.176643 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:33.176337    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:20:33.306692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:33.305607    5886 generic.go:356] "Generic (PLEG): container finished" podID="7c4b7e15-3adb-410b-b1da-a089490823af" containerID="10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63" exitCode=0
Oct 30 21:20:33.306692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:33.305759    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" event={"ID":"7c4b7e15-3adb-410b-b1da-a089490823af","Type":"ContainerDied","Data":"10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63"}
Oct 30 21:20:33.306692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:33.306105    5886 scope.go:122] "RemoveContainer" containerID="10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63"
Oct 30 21:20:33.307961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:33.307922    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:33.308053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:33.308019    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:33.308089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:33.308066    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:34.134500 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:34.134466    5886 patch_prober.go:29] interesting pod/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:20:34.134683 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:34.134518    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:20:34.328745 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:34.328708    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" event={"ID":"7c4b7e15-3adb-410b-b1da-a089490823af","Type":"ContainerStarted","Data":"2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a"}
Oct 30 21:20:34.352842 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:34.352807    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:34.353035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:34.353001    5886 patch_prober.go:29] interesting pod/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:20:34.353097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:34.353043    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:20:35.353647 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:35.353613    5886 patch_prober.go:29] interesting pod/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:20:35.354029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:35.353661    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:20:36.225129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:36.224435    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:36.310248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:36.309311    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:36.353376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:36.353338    5886 patch_prober.go:29] interesting pod/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:20:36.353565 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:36.353400    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:20:37.353515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:37.353173    5886 patch_prober.go:29] interesting pod/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:20:37.353515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:37.353228    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:20:38.353822 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:38.353144    5886 patch_prober.go:29] interesting pod/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:20:38.353822 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:38.353249    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:20:39.353755 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:39.353657    5886 patch_prober.go:29] interesting pod/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:20:39.353755 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:39.353724    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:20:39.387509 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:39.387456    5886 generic.go:356] "Generic (PLEG): container finished" podID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerID="313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd" exitCode=0
Oct 30 21:20:39.387706 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:39.387688    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" event={"ID":"0673bbbf-2ea9-47f4-918d-3b17ad723cdd","Type":"ContainerDied","Data":"313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd"}
Oct 30 21:20:39.388214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:39.388189    5886 scope.go:122] "RemoveContainer" containerID="313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd"
Oct 30 21:20:39.389869 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:39.389830    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:39.389944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:39.389926    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:39.390231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:39.389976    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:40.357462 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:40.357428    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:40.357917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:40.357899    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:40.358494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:40.358473    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:40.400885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:40.400297    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" event={"ID":"0673bbbf-2ea9-47f4-918d-3b17ad723cdd","Type":"ContainerStarted","Data":"bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa"}
Oct 30 21:20:40.884631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:40.884534    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:40.884791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:40.884705    5886 patch_prober.go:29] interesting pod/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:20:40.884791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:40.884752    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:20:41.034810 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:41.034773    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:20:41.878996 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:41.878940    5886 patch_prober.go:29] interesting pod/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:20:41.879800 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:41.879766    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:20:42.887671 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:42.887557    5886 patch_prober.go:29] interesting pod/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:20:42.887671 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:42.887635    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:20:43.422172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:43.422123    5886 generic.go:356] "Generic (PLEG): container finished" podID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerID="4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60" exitCode=0
Oct 30 21:20:43.422394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:43.422205    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" event={"ID":"82a50620-2814-4dc4-a8c9-493a6d4ea03a","Type":"ContainerDied","Data":"4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60"}
Oct 30 21:20:43.422640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:43.422624    5886 scope.go:122] "RemoveContainer" containerID="4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60"
Oct 30 21:20:43.424042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:43.424013    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:43.424183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:43.424091    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:43.424183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:43.424123    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:43.882039 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:43.881513    5886 patch_prober.go:29] interesting pod/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:20:43.882039 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:43.881576    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:20:44.437023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:44.436979    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" event={"ID":"82a50620-2814-4dc4-a8c9-493a6d4ea03a","Type":"ContainerStarted","Data":"bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42"}
Oct 30 21:20:44.887231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:44.886559    5886 patch_prober.go:29] interesting pod/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:20:44.887231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:44.886623    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:20:45.186276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:45.186230    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:45.198409 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:45.198358    5886 patch_prober.go:29] interesting pod/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:20:45.198731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:45.198685    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:20:45.903220 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:45.903166    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"]
Oct 30 21:20:45.917130 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:45.917095    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:45.917768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:45.917741    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:45.918710 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:45.918684    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:46.187812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:46.187710    5886 patch_prober.go:29] interesting pod/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:20:46.187812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:46.187770    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:20:47.189006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:47.188540    5886 patch_prober.go:29] interesting pod/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:20:47.189006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:47.188600    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:20:48.183594 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:48.183555    5886 patch_prober.go:29] interesting pod/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:20:48.183889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:48.183859    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:20:49.138487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:49.138452    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:49.184179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:49.183537    5886 patch_prober.go:29] interesting pod/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:20:49.184179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:49.183611    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 30 21:20:50.188448 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:50.188419    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:50.189685 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:50.189658    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:20:50.903543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:50.903498    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n" podUID="f3485681-3d15-4267-911b-692beabee729" containerName="payload-pre-processing" containerID="cri-o://ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c" gracePeriod=25
Oct 30 21:20:50.937443 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:50.937414    5886 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3485681_3d15_4267_911b_692beabee729.slice/crio-ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:20:50.996478 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:50.996447    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"]
Oct 30 21:20:51.181310 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.181281    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:51.260698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.260666    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3485681-3d15-4267-911b-692beabee729-kube-api-access-f67bb\" (UniqueName: \"kubernetes.io/projected/f3485681-3d15-4267-911b-692beabee729-kube-api-access-f67bb\") pod \"f3485681-3d15-4267-911b-692beabee729\" (UID: \"f3485681-3d15-4267-911b-692beabee729\") "
Oct 30 21:20:51.261178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.260743    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3485681-3d15-4267-911b-692beabee729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3485681-3d15-4267-911b-692beabee729-plugins-config-volume\") pod \"f3485681-3d15-4267-911b-692beabee729\" (UID: \"f3485681-3d15-4267-911b-692beabee729\") "
Oct 30 21:20:51.261252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.261220    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3485681-3d15-4267-911b-692beabee729-plugins-config-volume" pod "f3485681-3d15-4267-911b-692beabee729" (UID: "f3485681-3d15-4267-911b-692beabee729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:51.263188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.263146    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3485681-3d15-4267-911b-692beabee729-kube-api-access-f67bb" pod "f3485681-3d15-4267-911b-692beabee729" (UID: "f3485681-3d15-4267-911b-692beabee729"). InnerVolumeSpecName "kube-api-access-f67bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:51.362045 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.362003    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67bb\" (UniqueName: \"kubernetes.io/projected/f3485681-3d15-4267-911b-692beabee729-kube-api-access-f67bb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:51.362045 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.362038    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3485681-3d15-4267-911b-692beabee729-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:51.482715 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.482617    5886 generic.go:356] "Generic (PLEG): container finished" podID="f3485681-3d15-4267-911b-692beabee729" containerID="ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c" exitCode=0
Oct 30 21:20:51.482715 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.482675    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"
Oct 30 21:20:51.482715 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.482706    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n" event={"ID":"f3485681-3d15-4267-911b-692beabee729","Type":"ContainerDied","Data":"ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c"}
Oct 30 21:20:51.482993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.482756    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n" event={"ID":"f3485681-3d15-4267-911b-692beabee729","Type":"ContainerDied","Data":"088837a0f7cb60dbe67c3529cde0f441bce432f8bae572e07d58a58861e77bfc"}
Oct 30 21:20:51.482993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.482780    5886 scope.go:122] "RemoveContainer" containerID="ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c"
Oct 30 21:20:51.493226 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.493182    5886 scope.go:122] "RemoveContainer" containerID="ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c"
Oct 30 21:20:51.493553 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:51.493523    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c\": container with ID starting with ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c not found: ID does not exist" containerID="ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c"
Oct 30 21:20:51.493648 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.493562    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c"} err="failed to get container status \"ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c\": rpc error: code = NotFound desc = could not find container \"ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c\": container with ID starting with ebfdfe896347e6b937fc9df6d57f3133e2caa17dd5694ab171d7683cdc40b40c not found: ID does not exist"
Oct 30 21:20:51.531600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.531567    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"]
Oct 30 21:20:51.542724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.542698    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bb245e02-d7f5bcd79-gmb7n"]
Oct 30 21:20:51.672277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.665968    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"]
Oct 30 21:20:51.672277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.666242    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq" podUID="76961f99-09f5-45df-a9b7-156dcd5008e8" containerName="maas-api" containerID="cri-o://1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20" gracePeriod=30
Oct 30 21:20:51.992418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.992385    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3485681-3d15-4267-911b-692beabee729" path="/var/lib/kubelet/pods/f3485681-3d15-4267-911b-692beabee729/volumes"
Oct 30 21:20:51.998844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.998816    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"]
Oct 30 21:20:51.999050 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:51.999026    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" containerID="cri-o://2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a" gracePeriod=30
Oct 30 21:20:52.011348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.011323    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:52.071628 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.071545    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76961f99-09f5-45df-a9b7-156dcd5008e8-kube-api-access-9gg5d\" (UniqueName: \"kubernetes.io/projected/76961f99-09f5-45df-a9b7-156dcd5008e8-kube-api-access-9gg5d\") pod \"76961f99-09f5-45df-a9b7-156dcd5008e8\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") "
Oct 30 21:20:52.071628 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.071604    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-metrics-certs\") pod \"76961f99-09f5-45df-a9b7-156dcd5008e8\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") "
Oct 30 21:20:52.071837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.071630    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-maas-api-tls\") pod \"76961f99-09f5-45df-a9b7-156dcd5008e8\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") "
Oct 30 21:20:52.071837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.071698    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76961f99-09f5-45df-a9b7-156dcd5008e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76961f99-09f5-45df-a9b7-156dcd5008e8-openshift-service-ca\") pod \"76961f99-09f5-45df-a9b7-156dcd5008e8\" (UID: \"76961f99-09f5-45df-a9b7-156dcd5008e8\") "
Oct 30 21:20:52.072179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.072134    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76961f99-09f5-45df-a9b7-156dcd5008e8-openshift-service-ca" pod "76961f99-09f5-45df-a9b7-156dcd5008e8" (UID: "76961f99-09f5-45df-a9b7-156dcd5008e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:52.074006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.072741    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76961f99-09f5-45df-a9b7-156dcd5008e8-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:52.082882 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.082537    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-maas-api-tls" pod "76961f99-09f5-45df-a9b7-156dcd5008e8" (UID: "76961f99-09f5-45df-a9b7-156dcd5008e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:52.083014 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.082919    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76961f99-09f5-45df-a9b7-156dcd5008e8-kube-api-access-9gg5d" pod "76961f99-09f5-45df-a9b7-156dcd5008e8" (UID: "76961f99-09f5-45df-a9b7-156dcd5008e8"). InnerVolumeSpecName "kube-api-access-9gg5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:52.083014 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.082854    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-metrics-certs" pod "76961f99-09f5-45df-a9b7-156dcd5008e8" (UID: "76961f99-09f5-45df-a9b7-156dcd5008e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:52.173942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.173907    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gg5d\" (UniqueName: \"kubernetes.io/projected/76961f99-09f5-45df-a9b7-156dcd5008e8-kube-api-access-9gg5d\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:52.173942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.173935    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:52.173942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.173947    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76961f99-09f5-45df-a9b7-156dcd5008e8-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:52.489945 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.489910    5886 generic.go:356] "Generic (PLEG): container finished" podID="76961f99-09f5-45df-a9b7-156dcd5008e8" containerID="1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20" exitCode=0
Oct 30 21:20:52.490558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.489972    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"
Oct 30 21:20:52.490558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.489996    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq" event={"ID":"76961f99-09f5-45df-a9b7-156dcd5008e8","Type":"ContainerDied","Data":"1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20"}
Oct 30 21:20:52.490558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.490085    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq" event={"ID":"76961f99-09f5-45df-a9b7-156dcd5008e8","Type":"ContainerDied","Data":"04bfb47dc49de999d8af3f3810d0cdc0401a479fe2e9ab10354024717da9c280"}
Oct 30 21:20:52.490558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.490122    5886 scope.go:122] "RemoveContainer" containerID="1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20"
Oct 30 21:20:52.500323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.500308    5886 scope.go:122] "RemoveContainer" containerID="1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20"
Oct 30 21:20:52.500603 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:52.500583    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20\": container with ID starting with 1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20 not found: ID does not exist" containerID="1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20"
Oct 30 21:20:52.500651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.500611    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20"} err="failed to get container status \"1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20\": rpc error: code = NotFound desc = could not find container \"1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20\": container with ID starting with 1cf82038ac8d05fe0dbdff588cfa15caf0440571f4be00f00de018f4f7d07c20 not found: ID does not exist"
Oct 30 21:20:52.543666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.543639    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"]
Oct 30 21:20:52.549451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.549429    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bdcfca55-5675b8fc57-8j7sq"]
Oct 30 21:20:52.666088 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.666053    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"]
Oct 30 21:20:52.666350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:52.666321    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy" containerID="cri-o://bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa" gracePeriod=30
Oct 30 21:20:53.955625 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.955583    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"]
Oct 30 21:20:53.956287 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.956258    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f3485681-3d15-4267-911b-692beabee729" containerName="payload-pre-processing"
Oct 30 21:20:53.956287 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.956282    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3485681-3d15-4267-911b-692beabee729" containerName="payload-pre-processing"
Oct 30 21:20:53.956466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.956296    5886 cpu_manager.go:414] "containerMap: removing container" podUID="76961f99-09f5-45df-a9b7-156dcd5008e8" containerName="maas-api"
Oct 30 21:20:53.956466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.956304    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76961f99-09f5-45df-a9b7-156dcd5008e8" containerName="maas-api"
Oct 30 21:20:53.956466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.956404    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3485681-3d15-4267-911b-692beabee729" containerName="payload-pre-processing"
Oct 30 21:20:53.956466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.956417    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="76961f99-09f5-45df-a9b7-156dcd5008e8" containerName="maas-api"
Oct 30 21:20:53.961342 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.961317    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.977173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.977140    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-50ab77-openshift-default-dockercfg-tt79b\""
Oct 30 21:20:53.988597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.988571    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76961f99-09f5-45df-a9b7-156dcd5008e8" path="/var/lib/kubelet/pods/76961f99-09f5-45df-a9b7-156dcd5008e8/volumes"
Oct 30 21:20:53.992865 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.992840    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"]
Oct 30 21:20:53.993237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993216    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.993299 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993257    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.993337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993298    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.993472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993449    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2930b56b-c0fe-46a3-a059-04de254fb9f8-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.993528 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993496    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.993584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993563    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.993657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993637    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.993720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993700    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:53.993802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:53.993745    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fx27\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-kube-api-access-7fx27\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.094654    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.094761    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2930b56b-c0fe-46a3-a059-04de254fb9f8-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096113    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2930b56b-c0fe-46a3-a059-04de254fb9f8-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096204    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096463    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096508    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096533    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096625    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096671    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fx27\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-kube-api-access-7fx27\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096772    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096795    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096827    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.097121    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.096937    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.101209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.100575    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.122303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.119805    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.122567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.122544    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.122723 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.122699    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fx27\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-kube-api-access-7fx27\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.277503 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.277421    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:54.604734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.604648    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"]
Oct 30 21:20:54.608593 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:54.608565    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2930b56b_c0fe_46a3_a059_04de254fb9f8.slice/crio-87ca7aee26aa418ea6513bffab860bf9376a2d49d2b27f3246f76b53259e0240 WatchSource:0}: Error finding container 87ca7aee26aa418ea6513bffab860bf9376a2d49d2b27f3246f76b53259e0240: Status 404 returned error can't find the container with id 87ca7aee26aa418ea6513bffab860bf9376a2d49d2b27f3246f76b53259e0240
Oct 30 21:20:54.610925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.610893    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:54.611001 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.610957    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:54.611001 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:54.610986    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:55.360382 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:55.360350    5886 patch_prober.go:29] interesting pod/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:20:55.360835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:55.360406    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:20:55.516705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:55.516664    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" event={"ID":"2930b56b-c0fe-46a3-a059-04de254fb9f8","Type":"ContainerStarted","Data":"e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9"}
Oct 30 21:20:55.516892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:55.516730    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" event={"ID":"2930b56b-c0fe-46a3-a059-04de254fb9f8","Type":"ContainerStarted","Data":"87ca7aee26aa418ea6513bffab860bf9376a2d49d2b27f3246f76b53259e0240"}
Oct 30 21:20:55.586525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:55.586477    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" podStartSLOduration=2.586463034 podStartE2EDuration="2.586463034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:55.575181713 +0000 UTC m=+655.704433164" watchObservedRunningTime="2026-10-30 21:20:55.586463034 +0000 UTC m=+655.715714490"
Oct 30 21:20:55.707591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:55.707557    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"]
Oct 30 21:20:55.707845 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:55.707813    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy" containerID="cri-o://bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42" gracePeriod=30
Oct 30 21:20:55.997528 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:55.997412    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz" podUID="55ea7856-ad34-4014-8e37-6bb81955a2a9" containerName="payload-processing" containerID="cri-o://0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f" gracePeriod=25
Oct 30 21:20:56.268392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.268360    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:56.278588 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.278565    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:56.284220 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.284198    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:56.326124 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.326092    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ea7856-ad34-4014-8e37-6bb81955a2a9-kube-api-access-fd7b9\" (UniqueName: \"kubernetes.io/projected/55ea7856-ad34-4014-8e37-6bb81955a2a9-kube-api-access-fd7b9\") pod \"55ea7856-ad34-4014-8e37-6bb81955a2a9\" (UID: \"55ea7856-ad34-4014-8e37-6bb81955a2a9\") "
Oct 30 21:20:56.326324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.326281    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ea7856-ad34-4014-8e37-6bb81955a2a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ea7856-ad34-4014-8e37-6bb81955a2a9-plugins-config-volume\") pod \"55ea7856-ad34-4014-8e37-6bb81955a2a9\" (UID: \"55ea7856-ad34-4014-8e37-6bb81955a2a9\") "
Oct 30 21:20:56.326670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.326645    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ea7856-ad34-4014-8e37-6bb81955a2a9-plugins-config-volume" pod "55ea7856-ad34-4014-8e37-6bb81955a2a9" (UID: "55ea7856-ad34-4014-8e37-6bb81955a2a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:56.328457 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.328433    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ea7856-ad34-4014-8e37-6bb81955a2a9-kube-api-access-fd7b9" pod "55ea7856-ad34-4014-8e37-6bb81955a2a9" (UID: "55ea7856-ad34-4014-8e37-6bb81955a2a9"). InnerVolumeSpecName "kube-api-access-fd7b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:56.427519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.427484    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd7b9\" (UniqueName: \"kubernetes.io/projected/55ea7856-ad34-4014-8e37-6bb81955a2a9-kube-api-access-fd7b9\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:56.427519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.427512    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55ea7856-ad34-4014-8e37-6bb81955a2a9-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:56.524086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.523999    5886 generic.go:356] "Generic (PLEG): container finished" podID="55ea7856-ad34-4014-8e37-6bb81955a2a9" containerID="0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f" exitCode=0
Oct 30 21:20:56.524086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.524052    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"
Oct 30 21:20:56.524312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.524082    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz" event={"ID":"55ea7856-ad34-4014-8e37-6bb81955a2a9","Type":"ContainerDied","Data":"0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f"}
Oct 30 21:20:56.524312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.524134    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz" event={"ID":"55ea7856-ad34-4014-8e37-6bb81955a2a9","Type":"ContainerDied","Data":"f5aa4bed3782d2e9bc2b3fb1dc7fb9b6dd52644c7c36010444a05383b19e3d99"}
Oct 30 21:20:56.524312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.524184    5886 scope.go:122] "RemoveContainer" containerID="0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f"
Oct 30 21:20:56.524568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.524550    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:56.525653 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.525627    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:20:56.533845 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.533829    5886 scope.go:122] "RemoveContainer" containerID="0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f"
Oct 30 21:20:56.534114 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:56.534096    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f\": container with ID starting with 0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f not found: ID does not exist" containerID="0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f"
Oct 30 21:20:56.534191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.534122    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f"} err="failed to get container status \"0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f\": rpc error: code = NotFound desc = could not find container \"0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f\": container with ID starting with 0f2d1813f7fd50c36b31a5c15b07f626293ef0fcdb028378d1c58307fb75df0f not found: ID does not exist"
Oct 30 21:20:56.614706 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.614662    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"]
Oct 30 21:20:56.620102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.620077    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz"]
Oct 30 21:20:56.758386 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.758348    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"]
Oct 30 21:20:56.758657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:56.758601    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc" podUID="ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" containerName="maas-api" containerID="cri-o://df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1" gracePeriod=30
Oct 30 21:20:57.079419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.079349    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:57.134121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.134094    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-maas-api-tls\") pod \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") "
Oct 30 21:20:57.134477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.134200    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-openshift-service-ca\") pod \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") "
Oct 30 21:20:57.134477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.134298    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-kube-api-access-9rvcg\" (UniqueName: \"kubernetes.io/projected/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-kube-api-access-9rvcg\") pod \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") "
Oct 30 21:20:57.134477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.134386    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-metrics-certs\") pod \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\" (UID: \"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6\") "
Oct 30 21:20:57.134672 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.134602    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-openshift-service-ca" pod "ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" (UID: "ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:57.134817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.134793    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.136289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.136261    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-maas-api-tls" pod "ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" (UID: "ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:57.136477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.136456    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-metrics-certs" pod "ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" (UID: "ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:57.136837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.136801    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-kube-api-access-9rvcg" pod "ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" (UID: "ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6"). InnerVolumeSpecName "kube-api-access-9rvcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:57.223029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.222565    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-b72f7bd8-58dcdc997f-m78jz" podUID="55ea7856-ad34-4014-8e37-6bb81955a2a9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.52:50052\" within 1s: context deadline exceeded"
Oct 30 21:20:57.223029 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:57.222980    5886 logging.go:55] [core] [Channel #110 SubChannel #111]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.52:50052", ServerName: "10.133.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.52:50052: operation was canceled"
Oct 30 21:20:57.235894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.235853    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.235894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.235888    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvcg\" (UniqueName: \"kubernetes.io/projected/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-kube-api-access-9rvcg\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.236126 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.235903    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.300727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.300694    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:57.437364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437330    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-kube-api-access-gwfhj\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-kube-api-access-gwfhj\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.437852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437393    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c4b7e15-3adb-410b-b1da-a089490823af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4b7e15-3adb-410b-b1da-a089490823af-istio-podinfo\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.437852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437431    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-socket\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.437852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437467    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c4b7e15-3adb-410b-b1da-a089490823af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4b7e15-3adb-410b-b1da-a089490823af-istiod-ca-cert\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.437852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437484    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-data\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.437852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437539    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-istio-token\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-istio-token\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.437852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437598    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-credential-socket\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.437852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437784    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-socket" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.438247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437791    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-certs\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.438247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437899    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-credential-socket" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.438247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437926    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-envoy\") pod \"7c4b7e15-3adb-410b-b1da-a089490823af\" (UID: \"7c4b7e15-3adb-410b-b1da-a089490823af\") "
Oct 30 21:20:57.438247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.437984    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-certs" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.438247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.438010    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-data" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.438247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.438192    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4b7e15-3adb-410b-b1da-a089490823af-istiod-ca-cert" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:57.438561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.438404    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.438561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.438424    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4b7e15-3adb-410b-b1da-a089490823af-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.438561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.438439    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.438561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.438454    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.438561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.438468    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.439763 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.439732    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-istio-token" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:57.439882 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.439825    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-kube-api-access-gwfhj" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "kube-api-access-gwfhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:57.440016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.439993    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-envoy" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.440052 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.440025    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c4b7e15-3adb-410b-b1da-a089490823af-istio-podinfo" pod "7c4b7e15-3adb-410b-b1da-a089490823af" (UID: "7c4b7e15-3adb-410b-b1da-a089490823af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:20:57.535688 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.535654    5886 generic.go:356] "Generic (PLEG): container finished" podID="7c4b7e15-3adb-410b-b1da-a089490823af" containerID="2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a" exitCode=0
Oct 30 21:20:57.535888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.535702    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"
Oct 30 21:20:57.535888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.535735    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" event={"ID":"7c4b7e15-3adb-410b-b1da-a089490823af","Type":"ContainerDied","Data":"2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a"}
Oct 30 21:20:57.535888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.535788    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt" event={"ID":"7c4b7e15-3adb-410b-b1da-a089490823af","Type":"ContainerDied","Data":"352e7f6fb2701a1eaf861c2804208e11786b6023748a846b824993c234d2dcdd"}
Oct 30 21:20:57.535888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.535804    5886 scope.go:122] "RemoveContainer" containerID="2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a"
Oct 30 21:20:57.541668 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.541643    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4b7e15-3adb-410b-b1da-a089490823af-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.541781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.541672    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwfhj\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-kube-api-access-gwfhj\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.541781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.541688    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4b7e15-3adb-410b-b1da-a089490823af-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.541781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.541702    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4b7e15-3adb-410b-b1da-a089490823af-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:57.541919 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.541898    5886 generic.go:356] "Generic (PLEG): container finished" podID="ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" containerID="df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1" exitCode=0
Oct 30 21:20:57.541962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.541938    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"
Oct 30 21:20:57.542025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.541980    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc" event={"ID":"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6","Type":"ContainerDied","Data":"df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1"}
Oct 30 21:20:57.542076 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.542040    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc" event={"ID":"ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6","Type":"ContainerDied","Data":"554a06355672422595918aecfe0adc5810b72149942bfa13622cef12c4be39ad"}
Oct 30 21:20:57.546452 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.546433    5886 scope.go:122] "RemoveContainer" containerID="10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63"
Oct 30 21:20:57.555770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.555754    5886 scope.go:122] "RemoveContainer" containerID="2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a"
Oct 30 21:20:57.556019 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:57.556000    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a\": container with ID starting with 2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a not found: ID does not exist" containerID="2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a"
Oct 30 21:20:57.556069 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.556025    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a"} err="failed to get container status \"2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a\": rpc error: code = NotFound desc = could not find container \"2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a\": container with ID starting with 2fd439d1d3ad190acd8bb95035fe73562aaff146e9d7ebb85b4d5f1ea0dbdf9a not found: ID does not exist"
Oct 30 21:20:57.556069 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.556041    5886 scope.go:122] "RemoveContainer" containerID="10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63"
Oct 30 21:20:57.556319 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:57.556300    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63\": container with ID starting with 10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63 not found: ID does not exist" containerID="10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63"
Oct 30 21:20:57.556397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.556330    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63"} err="failed to get container status \"10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63\": rpc error: code = NotFound desc = could not find container \"10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63\": container with ID starting with 10ef7a34e636604ad43ddbdaf45bf53011aeae8aea2774b84158458ed04a9c63 not found: ID does not exist"
Oct 30 21:20:57.556397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.556353    5886 scope.go:122] "RemoveContainer" containerID="df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1"
Oct 30 21:20:57.564848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.564833    5886 scope.go:122] "RemoveContainer" containerID="df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1"
Oct 30 21:20:57.565119 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:57.565087    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1\": container with ID starting with df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1 not found: ID does not exist" containerID="df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1"
Oct 30 21:20:57.565227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.565193    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1"} err="failed to get container status \"df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1\": rpc error: code = NotFound desc = could not find container \"df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1\": container with ID starting with df70cb3ab057ded5d39647d8c12fe20e9ba5cdfc8aa4fd8d655c905b152cdfa1 not found: ID does not exist"
Oct 30 21:20:57.596936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.596905    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"]
Oct 30 21:20:57.600818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.600793    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bb245e02-openshift-default-578ffddc7-jgnnt"]
Oct 30 21:20:57.633446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.633412    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"]
Oct 30 21:20:57.638818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.638792    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-acb99762-6467bf4c98-2rmnc"]
Oct 30 21:20:57.940427 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.940224    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:57.991607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.991389    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ea7856-ad34-4014-8e37-6bb81955a2a9" path="/var/lib/kubelet/pods/55ea7856-ad34-4014-8e37-6bb81955a2a9/volumes"
Oct 30 21:20:57.991988 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.991965    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" path="/var/lib/kubelet/pods/7c4b7e15-3adb-410b-b1da-a089490823af/volumes"
Oct 30 21:20:57.992530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:57.992502    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" path="/var/lib/kubelet/pods/ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6/volumes"
Oct 30 21:20:58.046751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.046721    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-podinfo\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.046899 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.046757    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-kube-api-access-hrcjq\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-kube-api-access-hrcjq\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.046899 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.046787    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-data\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.046981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.046916    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-credential-socket\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.047033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.046977    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-envoy\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.047033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047018    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-token\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-token\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.047136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047049    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-data" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:58.047136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047064    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istiod-ca-cert\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.047283 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047146    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-socket\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.047283 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047188    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-credential-socket" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:58.047283 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047235    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-certs\") pod \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\" (UID: \"0673bbbf-2ea9-47f4-918d-3b17ad723cdd\") "
Oct 30 21:20:58.047509 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047449    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-socket" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:58.047509 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047469    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-certs" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:58.047509 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047477    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istiod-ca-cert" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:58.047724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047699    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.047724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047727    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.047900 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047741    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.047900 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047755    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.047900 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.047768    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.049058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.049037    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-envoy" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:58.049141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.049082    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-podinfo" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:20:58.049141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.049123    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-kube-api-access-hrcjq" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "kube-api-access-hrcjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:58.049296 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.049279    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-token" pod "0673bbbf-2ea9-47f4-918d-3b17ad723cdd" (UID: "0673bbbf-2ea9-47f4-918d-3b17ad723cdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:58.148970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.148933    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.148970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.148961    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.148970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.148970    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.148970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.148979    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrcjq\" (UniqueName: \"kubernetes.io/projected/0673bbbf-2ea9-47f4-918d-3b17ad723cdd-kube-api-access-hrcjq\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:20:58.551335 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.551239    5886 generic.go:356] "Generic (PLEG): container finished" podID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerID="bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa" exitCode=0
Oct 30 21:20:58.551335 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.551317    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"
Oct 30 21:20:58.551849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.551316    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" event={"ID":"0673bbbf-2ea9-47f4-918d-3b17ad723cdd","Type":"ContainerDied","Data":"bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa"}
Oct 30 21:20:58.551849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.551536    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98" event={"ID":"0673bbbf-2ea9-47f4-918d-3b17ad723cdd","Type":"ContainerDied","Data":"2a71732b8ab6d5ba15691203aaa4c9a3c19c9db85949828e5ea088b25d696a57"}
Oct 30 21:20:58.551849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.551576    5886 scope.go:122] "RemoveContainer" containerID="bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa"
Oct 30 21:20:58.562248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.562228    5886 scope.go:122] "RemoveContainer" containerID="313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd"
Oct 30 21:20:58.572680 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.572664    5886 scope.go:122] "RemoveContainer" containerID="bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa"
Oct 30 21:20:58.573006 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:58.572981    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa\": container with ID starting with bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa not found: ID does not exist" containerID="bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa"
Oct 30 21:20:58.573072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.573014    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa"} err="failed to get container status \"bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa\": rpc error: code = NotFound desc = could not find container \"bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa\": container with ID starting with bf65ea3b175ee505cf0d86a4dbda96ebd4b5e66b2db145598c15e7117cdfeefa not found: ID does not exist"
Oct 30 21:20:58.573072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.573029    5886 scope.go:122] "RemoveContainer" containerID="313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd"
Oct 30 21:20:58.573338 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:20:58.573322    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd\": container with ID starting with 313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd not found: ID does not exist" containerID="313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd"
Oct 30 21:20:58.573401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.573341    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd"} err="failed to get container status \"313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd\": rpc error: code = NotFound desc = could not find container \"313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd\": container with ID starting with 313e2f07dd044ca49738f6666e122f081dec442791955908137363f1fe12cebd not found: ID does not exist"
Oct 30 21:20:58.620426 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.620393    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"]
Oct 30 21:20:58.631271 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.631246    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bdcfca55-gw-openshift-default-57c48fc5df-hks98"]
Oct 30 21:20:58.744278 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.743899    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"]
Oct 30 21:20:58.744779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744711    5886 cpu_manager.go:414] "containerMap: removing container" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy"
Oct 30 21:20:58.744779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744734    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy"
Oct 30 21:20:58.744779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744754    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy"
Oct 30 21:20:58.744779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744761    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy"
Oct 30 21:20:58.744779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744771    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" containerName="maas-api"
Oct 30 21:20:58.744779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744779    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" containerName="maas-api"
Oct 30 21:20:58.744779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744789    5886 cpu_manager.go:414] "containerMap: removing container" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744794    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744801    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744805    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744815    5886 cpu_manager.go:414] "containerMap: removing container" podUID="55ea7856-ad34-4014-8e37-6bb81955a2a9" containerName="payload-processing"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744822    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ea7856-ad34-4014-8e37-6bb81955a2a9" containerName="payload-processing"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744907    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744915    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744920    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb7dfa2-8f6e-4c9e-9ddb-79e6f55848a6" containerName="maas-api"
Oct 30 21:20:58.745082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.744927    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ea7856-ad34-4014-8e37-6bb81955a2a9" containerName="payload-processing"
Oct 30 21:20:58.749748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.749729    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"]
Oct 30 21:20:58.749858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.749839    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:58.750331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.750312    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" containerName="istio-proxy"
Oct 30 21:20:58.750331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.750330    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4b7e15-3adb-410b-b1da-a089490823af" containerName="istio-proxy"
Oct 30 21:20:58.753343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.753323    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:58.766971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.766951    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ea40ee\""
Oct 30 21:20:58.767968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.767954    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ea40ee-dockercfg-mj2mk\""
Oct 30 21:20:58.771011 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.770952    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"]
Oct 30 21:20:58.773911 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.773887    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"]
Oct 30 21:20:58.855288 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.855193    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznxj\" (UniqueName: \"kubernetes.io/projected/6ed04681-8f10-4915-85ba-d0df6be22c13-kube-api-access-rznxj\") pod \"payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6\" (UID: \"6ed04681-8f10-4915-85ba-d0df6be22c13\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:58.855288 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.855276    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458xf\" (UniqueName: \"kubernetes.io/projected/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-kube-api-access-458xf\") pod \"payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2\" (UID: \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:58.855521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.855315    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2\" (UID: \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:58.855521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.855357    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed04681-8f10-4915-85ba-d0df6be22c13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6\" (UID: \"6ed04681-8f10-4915-85ba-d0df6be22c13\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:58.956800 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.956766    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznxj\" (UniqueName: \"kubernetes.io/projected/6ed04681-8f10-4915-85ba-d0df6be22c13-kube-api-access-rznxj\") pod \"payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6\" (UID: \"6ed04681-8f10-4915-85ba-d0df6be22c13\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:58.956966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.956822    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458xf\" (UniqueName: \"kubernetes.io/projected/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-kube-api-access-458xf\") pod \"payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2\" (UID: \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:58.956966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.956847    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2\" (UID: \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:58.956966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.956874    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed04681-8f10-4915-85ba-d0df6be22c13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6\" (UID: \"6ed04681-8f10-4915-85ba-d0df6be22c13\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:58.957623 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.957602    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed04681-8f10-4915-85ba-d0df6be22c13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6\" (UID: \"6ed04681-8f10-4915-85ba-d0df6be22c13\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:58.957690 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.957665    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2\" (UID: \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:58.976755 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.976725    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458xf\" (UniqueName: \"kubernetes.io/projected/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-kube-api-access-458xf\") pod \"payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2\" (UID: \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:58.977559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:58.977537    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznxj\" (UniqueName: \"kubernetes.io/projected/6ed04681-8f10-4915-85ba-d0df6be22c13-kube-api-access-rznxj\") pod \"payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6\" (UID: \"6ed04681-8f10-4915-85ba-d0df6be22c13\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:59.064329 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.064292    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:59.070520 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.070496    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:59.285382 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.285347    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"]
Oct 30 21:20:59.290920 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.290895    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.309915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.309895    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9d677d-openshift-default-dockercfg-l57n8\""
Oct 30 21:20:59.325953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.325929    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"]
Oct 30 21:20:59.360310 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360272    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.360487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360320    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.360487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360433    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.360487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360476    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.360587 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360548    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.360649 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360633    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv92z\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-kube-api-access-mv92z\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.360705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360691    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.360783 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360758    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.360852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.360816    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.381706 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.381639    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"]
Oct 30 21:20:59.385974 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:59.385952    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee44288_11ec_48fd_aa6f_8f83e27b27a1.slice/crio-969f6f70361ed04f560ba9d2ad8ba7015caedf3eacb710df1c2ae369aa9dee05 WatchSource:0}: Error finding container 969f6f70361ed04f560ba9d2ad8ba7015caedf3eacb710df1c2ae369aa9dee05: Status 404 returned error can't find the container with id 969f6f70361ed04f560ba9d2ad8ba7015caedf3eacb710df1c2ae369aa9dee05
Oct 30 21:20:59.404309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.404236    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"]
Oct 30 21:20:59.412744 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:59.412712    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed04681_8f10_4915_85ba_d0df6be22c13.slice/crio-4cb1a2b14cd88a6837433b6d7a9e504ec98e457c976607d15e23085942db8ac4 WatchSource:0}: Error finding container 4cb1a2b14cd88a6837433b6d7a9e504ec98e457c976607d15e23085942db8ac4: Status 404 returned error can't find the container with id 4cb1a2b14cd88a6837433b6d7a9e504ec98e457c976607d15e23085942db8ac4
Oct 30 21:20:59.462202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462170    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv92z\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-kube-api-access-mv92z\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462236    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462274    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462320    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462382    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462408    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462461    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462502    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462542    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462724    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462794    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.462934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462914    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.463011 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.462993    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.463210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.463183    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.464668 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.464647    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.464876 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.464858    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.485129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.485099    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv92z\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-kube-api-access-mv92z\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.486055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.486016    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.561328 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.561286    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6" event={"ID":"6ed04681-8f10-4915-85ba-d0df6be22c13","Type":"ContainerStarted","Data":"c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8"}
Oct 30 21:20:59.561752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.561354    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6" event={"ID":"6ed04681-8f10-4915-85ba-d0df6be22c13","Type":"ContainerStarted","Data":"4cb1a2b14cd88a6837433b6d7a9e504ec98e457c976607d15e23085942db8ac4"}
Oct 30 21:20:59.561752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.561386    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:20:59.563891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.563865    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2" event={"ID":"5ee44288-11ec-48fd-aa6f-8f83e27b27a1","Type":"ContainerStarted","Data":"416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb"}
Oct 30 21:20:59.564012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.563906    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2" event={"ID":"5ee44288-11ec-48fd-aa6f-8f83e27b27a1","Type":"ContainerStarted","Data":"969f6f70361ed04f560ba9d2ad8ba7015caedf3eacb710df1c2ae369aa9dee05"}
Oct 30 21:20:59.564012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.563941    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:20:59.601359 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.601304    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6" podStartSLOduration=1.601289416 podStartE2EDuration="1.601289416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:59.600416193 +0000 UTC m=+659.729667651" watchObservedRunningTime="2026-10-30 21:20:59.601289416 +0000 UTC m=+659.730540873"
Oct 30 21:20:59.607551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.607521    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:20:59.639051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.638971    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2" podStartSLOduration=1.6389583170000002 podStartE2EDuration="1.638958317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:59.638087262 +0000 UTC m=+659.767338721" watchObservedRunningTime="2026-10-30 21:20:59.638958317 +0000 UTC m=+659.768209774"
Oct 30 21:20:59.926442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.926397    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"]
Oct 30 21:20:59.931673 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:20:59.931643    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05864e5f_02e0_4bf2_b5b9_4130a34593dd.slice/crio-ade17f2c48f7f44e1102c63254710eab080c22132ba91fc25b4e48b008b851d2 WatchSource:0}: Error finding container ade17f2c48f7f44e1102c63254710eab080c22132ba91fc25b4e48b008b851d2: Status 404 returned error can't find the container with id ade17f2c48f7f44e1102c63254710eab080c22132ba91fc25b4e48b008b851d2
Oct 30 21:20:59.934231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.934120    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:59.934329 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.934232    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:59.934329 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.934263    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:20:59.998042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:20:59.998008    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0673bbbf-2ea9-47f4-918d-3b17ad723cdd" path="/var/lib/kubelet/pods/0673bbbf-2ea9-47f4-918d-3b17ad723cdd/volumes"
Oct 30 21:21:00.571013 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.570974    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" event={"ID":"05864e5f-02e0-4bf2-b5b9-4130a34593dd","Type":"ContainerStarted","Data":"a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68"}
Oct 30 21:21:00.571473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.571030    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" event={"ID":"05864e5f-02e0-4bf2-b5b9-4130a34593dd","Type":"ContainerStarted","Data":"ade17f2c48f7f44e1102c63254710eab080c22132ba91fc25b4e48b008b851d2"}
Oct 30 21:21:00.607805 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.607778    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:21:00.616545 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.616518    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"]
Oct 30 21:21:00.618895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.618849    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" podStartSLOduration=1.618828271 podStartE2EDuration="1.618828271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:00.617454377 +0000 UTC m=+660.746705834" watchObservedRunningTime="2026-10-30 21:21:00.618828271 +0000 UTC m=+660.748079727"
Oct 30 21:21:00.621907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.621888    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:00.635705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.635683    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-50ab77\""
Oct 30 21:21:00.635932 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.635912    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-50ab77-dockercfg-255j4\""
Oct 30 21:21:00.638655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.638587    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"]
Oct 30 21:21:00.673611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.673579    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017cb3c5-df91-42c3-9061-e5545a46a365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6\" (UID: \"017cb3c5-df91-42c3-9061-e5545a46a365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:00.673776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.673620    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn68b\" (UniqueName: \"kubernetes.io/projected/017cb3c5-df91-42c3-9061-e5545a46a365-kube-api-access-rn68b\") pod \"payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6\" (UID: \"017cb3c5-df91-42c3-9061-e5545a46a365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:00.706854 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.706807    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"]
Oct 30 21:21:00.710703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.710681    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:00.732408 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.732363    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"]
Oct 30 21:21:00.779204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.777741    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppxn\" (UniqueName: \"kubernetes.io/projected/11c5825c-9673-43e1-9316-9e98058a4fa4-kube-api-access-6ppxn\") pod \"payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn\" (UID: \"11c5825c-9673-43e1-9316-9e98058a4fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:00.779204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.777808    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017cb3c5-df91-42c3-9061-e5545a46a365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6\" (UID: \"017cb3c5-df91-42c3-9061-e5545a46a365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:00.779204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.777848    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn68b\" (UniqueName: \"kubernetes.io/projected/017cb3c5-df91-42c3-9061-e5545a46a365-kube-api-access-rn68b\") pod \"payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6\" (UID: \"017cb3c5-df91-42c3-9061-e5545a46a365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:00.779204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.777893    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c5825c-9673-43e1-9316-9e98058a4fa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn\" (UID: \"11c5825c-9673-43e1-9316-9e98058a4fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:00.787737 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.787703    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017cb3c5-df91-42c3-9061-e5545a46a365-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6\" (UID: \"017cb3c5-df91-42c3-9061-e5545a46a365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:00.808183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.808104    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn68b\" (UniqueName: \"kubernetes.io/projected/017cb3c5-df91-42c3-9061-e5545a46a365-kube-api-access-rn68b\") pod \"payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6\" (UID: \"017cb3c5-df91-42c3-9061-e5545a46a365\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:00.880958 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.880925    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppxn\" (UniqueName: \"kubernetes.io/projected/11c5825c-9673-43e1-9316-9e98058a4fa4-kube-api-access-6ppxn\") pod \"payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn\" (UID: \"11c5825c-9673-43e1-9316-9e98058a4fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:00.881247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.881215    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c5825c-9673-43e1-9316-9e98058a4fa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn\" (UID: \"11c5825c-9673-43e1-9316-9e98058a4fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:00.882234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.882209    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c5825c-9673-43e1-9316-9e98058a4fa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn\" (UID: \"11c5825c-9673-43e1-9316-9e98058a4fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:00.902085 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.902060    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppxn\" (UniqueName: \"kubernetes.io/projected/11c5825c-9673-43e1-9316-9e98058a4fa4-kube-api-access-6ppxn\") pod \"payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn\" (UID: \"11c5825c-9673-43e1-9316-9e98058a4fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:00.936928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:00.936892    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:01.019925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.017768    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:21:01.024317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.024292    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082490    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-podinfo\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082560    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-socket\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082601    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-envoy\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082649    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-credential-socket\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082687    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-certs\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082728    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-data\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082765    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-token\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-token\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082797    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istiod-ca-cert\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.082896    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-kube-api-access-xglkb\") pod \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\" (UID: \"82a50620-2814-4dc4-a8c9-493a6d4ea03a\") "
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.083184    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-certs" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.083334    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.083549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.083526    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-socket" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:01.087040 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.084362    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istiod-ca-cert" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:01.087040 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.084531    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-data" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:01.087040 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.084621    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-credential-socket" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:01.088761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.088732    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-podinfo" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:21:01.089176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.089117    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-envoy" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:01.089429 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.089394    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-kube-api-access-xglkb" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "kube-api-access-xglkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:01.094021 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.093980    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-token" pod "82a50620-2814-4dc4-a8c9-493a6d4ea03a" (UID: "82a50620-2814-4dc4-a8c9-493a6d4ea03a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:01.184501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.184468    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xglkb\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-kube-api-access-xglkb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.184501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.184494    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.184501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.184506    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.184803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.184515    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.184803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.184523    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.184803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.184532    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.184803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.184539    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.184803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.184546    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a50620-2814-4dc4-a8c9-493a6d4ea03a-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:21:01.253492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.253457    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"]
Oct 30 21:21:01.257570 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:21:01.257536    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017cb3c5_df91_42c3_9061_e5545a46a365.slice/crio-6d6b90ebe6cdd82c94570de9236f8d63558648c62e685008003db13027c2d8af WatchSource:0}: Error finding container 6d6b90ebe6cdd82c94570de9236f8d63558648c62e685008003db13027c2d8af: Status 404 returned error can't find the container with id 6d6b90ebe6cdd82c94570de9236f8d63558648c62e685008003db13027c2d8af
Oct 30 21:21:01.343008 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.342980    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"]
Oct 30 21:21:01.345613 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:21:01.345582    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c5825c_9673_43e1_9316_9e98058a4fa4.slice/crio-6e5f6e39a7ac018ad4be1a8da58bec26a90a225f1cb0aa2d0b7c7ed5373c34cb WatchSource:0}: Error finding container 6e5f6e39a7ac018ad4be1a8da58bec26a90a225f1cb0aa2d0b7c7ed5373c34cb: Status 404 returned error can't find the container with id 6e5f6e39a7ac018ad4be1a8da58bec26a90a225f1cb0aa2d0b7c7ed5373c34cb
Oct 30 21:21:01.533884 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.533696    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"]
Oct 30 21:21:01.534414 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.534394    5886 cpu_manager.go:414] "containerMap: removing container" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy"
Oct 30 21:21:01.534498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.534415    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy"
Oct 30 21:21:01.534498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.534426    5886 cpu_manager.go:414] "containerMap: removing container" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy"
Oct 30 21:21:01.534498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.534431    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy"
Oct 30 21:21:01.534596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.534530    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy"
Oct 30 21:21:01.534596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.534537    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerName="istio-proxy"
Oct 30 21:21:01.537928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.537907    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.556477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.556433    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"]
Oct 30 21:21:01.557356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.557128    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ea40ee-serving-cert\""
Oct 30 21:21:01.581968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.581927    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn" event={"ID":"11c5825c-9673-43e1-9316-9e98058a4fa4","Type":"ContainerStarted","Data":"90c04938f696adaf53c4ad0652e1062858ba661522ec42c3a6eba4553bbb1b2e"}
Oct 30 21:21:01.582527 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.581991    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn" event={"ID":"11c5825c-9673-43e1-9316-9e98058a4fa4","Type":"ContainerStarted","Data":"6e5f6e39a7ac018ad4be1a8da58bec26a90a225f1cb0aa2d0b7c7ed5373c34cb"}
Oct 30 21:21:01.582527 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.582335    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:01.584871 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.584844    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6" event={"ID":"017cb3c5-df91-42c3-9061-e5545a46a365","Type":"ContainerStarted","Data":"9277bd1731c897db71eb5524324ab09a90da18b1528d015b92548c3069617d0b"}
Oct 30 21:21:01.585045 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.585020    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6" event={"ID":"017cb3c5-df91-42c3-9061-e5545a46a365","Type":"ContainerStarted","Data":"6d6b90ebe6cdd82c94570de9236f8d63558648c62e685008003db13027c2d8af"}
Oct 30 21:21:01.585196 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.585061    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:01.586754 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.586731    5886 generic.go:356] "Generic (PLEG): container finished" podID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" containerID="bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42" exitCode=0
Oct 30 21:21:01.586870 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.586782    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"
Oct 30 21:21:01.586870 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.586848    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" event={"ID":"82a50620-2814-4dc4-a8c9-493a6d4ea03a","Type":"ContainerDied","Data":"bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42"}
Oct 30 21:21:01.586978 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.586893    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz" event={"ID":"82a50620-2814-4dc4-a8c9-493a6d4ea03a","Type":"ContainerDied","Data":"0f47c89216e40dd59bd38b722005940204e3669c2aa1d7fae6393c00809ceb5f"}
Oct 30 21:21:01.586978 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.586925    5886 scope.go:122] "RemoveContainer" containerID="bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42"
Oct 30 21:21:01.599088 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.599066    5886 scope.go:122] "RemoveContainer" containerID="4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60"
Oct 30 21:21:01.608517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.608497    5886 scope.go:122] "RemoveContainer" containerID="bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42"
Oct 30 21:21:01.608806 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:21:01.608782    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42\": container with ID starting with bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42 not found: ID does not exist" containerID="bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42"
Oct 30 21:21:01.608886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.608817    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42"} err="failed to get container status \"bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42\": rpc error: code = NotFound desc = could not find container \"bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42\": container with ID starting with bd20c712308214ccbcf303863391aaefece77f1efcae780dc85b5b863236fe42 not found: ID does not exist"
Oct 30 21:21:01.608886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.608840    5886 scope.go:122] "RemoveContainer" containerID="4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60"
Oct 30 21:21:01.609268 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:21:01.609136    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60\": container with ID starting with 4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60 not found: ID does not exist" containerID="4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60"
Oct 30 21:21:01.609341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.609281    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60"} err="failed to get container status \"4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60\": rpc error: code = NotFound desc = could not find container \"4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60\": container with ID starting with 4f936eb44f5b11034325304561fe4966eb987f76368f9672c5ef52bba9cd3e60 not found: ID does not exist"
Oct 30 21:21:01.613189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.613169    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:21:01.619917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.619873    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn" podStartSLOduration=1.6198612639999999 podStartE2EDuration="1.619861264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:01.618547181 +0000 UTC m=+661.747798639" watchObservedRunningTime="2026-10-30 21:21:01.619861264 +0000 UTC m=+661.749112718"
Oct 30 21:21:01.665358 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.665308    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6" podStartSLOduration=1.665293695 podStartE2EDuration="1.665293695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:01.662770658 +0000 UTC m=+661.792022115" watchObservedRunningTime="2026-10-30 21:21:01.665293695 +0000 UTC m=+661.794545149"
Oct 30 21:21:01.688837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.688807    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12524c2e-b65b-4b38-aea2-b0e7922e2d14-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.689016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.688977    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.689130 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.689110    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnpb\" (UniqueName: \"kubernetes.io/projected/12524c2e-b65b-4b38-aea2-b0e7922e2d14-kube-api-access-5dnpb\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.689363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.689334    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-metrics-certs\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.697741 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.697697    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"]
Oct 30 21:21:01.707442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.707415    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b72f7bd8-openshift-default-7bf84c4565-pg8bz"]
Oct 30 21:21:01.790577 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.790541    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.790751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.790585    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnpb\" (UniqueName: \"kubernetes.io/projected/12524c2e-b65b-4b38-aea2-b0e7922e2d14-kube-api-access-5dnpb\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.790751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.790645    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-metrics-certs\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.790879 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.790797    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12524c2e-b65b-4b38-aea2-b0e7922e2d14-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.791525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.791497    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12524c2e-b65b-4b38-aea2-b0e7922e2d14-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.792956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.792931    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.793049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.793010    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-metrics-certs\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.814032 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.813994    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnpb\" (UniqueName: \"kubernetes.io/projected/12524c2e-b65b-4b38-aea2-b0e7922e2d14-kube-api-access-5dnpb\") pod \"maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.852826 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.852652    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:01.992295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:01.992259    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a50620-2814-4dc4-a8c9-493a6d4ea03a" path="/var/lib/kubelet/pods/82a50620-2814-4dc4-a8c9-493a6d4ea03a/volumes"
Oct 30 21:21:02.197295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:02.197262    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"]
Oct 30 21:21:02.201799 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:21:02.201769    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12524c2e_b65b_4b38_aea2_b0e7922e2d14.slice/crio-124e4504be3750edbf5f73c6a9f63123c751e9d3897c5dcd8e6ba4eff10dfd82 WatchSource:0}: Error finding container 124e4504be3750edbf5f73c6a9f63123c751e9d3897c5dcd8e6ba4eff10dfd82: Status 404 returned error can't find the container with id 124e4504be3750edbf5f73c6a9f63123c751e9d3897c5dcd8e6ba4eff10dfd82
Oct 30 21:21:02.595000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:02.594969    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb" event={"ID":"12524c2e-b65b-4b38-aea2-b0e7922e2d14","Type":"ContainerStarted","Data":"124e4504be3750edbf5f73c6a9f63123c751e9d3897c5dcd8e6ba4eff10dfd82"}
Oct 30 21:21:02.596569 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:02.596537    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:21:02.598046 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:02.598009    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:21:03.578676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.578635    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"]
Oct 30 21:21:03.582935 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.582911    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.603341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.603287    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb" event={"ID":"12524c2e-b65b-4b38-aea2-b0e7922e2d14","Type":"ContainerStarted","Data":"62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b"}
Oct 30 21:21:03.603953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.603919    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:03.608605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.608580    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-50ab77-serving-cert\""
Oct 30 21:21:03.611117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.611095    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"]
Oct 30 21:21:03.672309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.672247    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb" podStartSLOduration=2.335929212 podStartE2EDuration="2.672229124s" totalImagesPullingTime="336.299912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:03.670271862 +0000 UTC m=+663.799523319" watchObservedRunningTime="2026-10-30 21:21:03.672229124 +0000 UTC m=+663.801480587"
Oct 30 21:21:03.708391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.708355    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6xzm\" (UniqueName: \"kubernetes.io/projected/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-kube-api-access-w6xzm\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.708560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.708419    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.708624 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.708581    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-metrics-certs\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.708821 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.708796    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.810326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.810298    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.810489 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.810350    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-metrics-certs\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.810489 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.810413    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.810575 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:21:03.810523    5886 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-serving-cert: secret "maas-api-e2e-worker-w3-50ab77-serving-cert" not found
Oct 30 21:21:03.810575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.810544    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6xzm\" (UniqueName: \"kubernetes.io/projected/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-kube-api-access-w6xzm\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.810658 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:21:03.810578    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls podName:01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e nodeName:}" failed. No retries permitted until 2026-10-30 21:21:04.310562678 +0000 UTC m=+664.439814113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls") pod "maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk" (UID: "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e") : secret "maas-api-e2e-worker-w3-50ab77-serving-cert" not found
Oct 30 21:21:03.811134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.811112    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.812967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.812946    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-metrics-certs\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:03.835564 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:03.835500    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6xzm\" (UniqueName: \"kubernetes.io/projected/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-kube-api-access-w6xzm\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:04.315460 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:04.315423    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:04.317735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:04.317708    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:04.496483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:04.496447    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:04.803439 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:04.803387    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"]
Oct 30 21:21:04.807856 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:21:04.807827    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ffc8a1_8d4a_41cf_be78_7c82a8e5ad2e.slice/crio-e58cefebd34f48e5827d17d043ad5519251fb841f77db41eb7ed9abee5128e98 WatchSource:0}: Error finding container e58cefebd34f48e5827d17d043ad5519251fb841f77db41eb7ed9abee5128e98: Status 404 returned error can't find the container with id e58cefebd34f48e5827d17d043ad5519251fb841f77db41eb7ed9abee5128e98
Oct 30 21:21:05.618257 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:05.618176    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk" event={"ID":"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e","Type":"ContainerStarted","Data":"b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697"}
Oct 30 21:21:05.618257 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:05.618229    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk" event={"ID":"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e","Type":"ContainerStarted","Data":"e58cefebd34f48e5827d17d043ad5519251fb841f77db41eb7ed9abee5128e98"}
Oct 30 21:21:05.618436 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:05.618315    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:05.658769 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:05.658714    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk" podStartSLOduration=2.276817966 podStartE2EDuration="2.658700231s" totalImagesPullingTime="381.882265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:05.656771951 +0000 UTC m=+665.786023408" watchObservedRunningTime="2026-10-30 21:21:05.658700231 +0000 UTC m=+665.787951687"
Oct 30 21:21:09.616989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:09.616957    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 21:21:10.587645 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:10.587612    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 21:21:10.587839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:10.587723    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 21:21:11.028561 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:21:11.028522    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:21:11.630222 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:11.630188    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 21:21:12.599717 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:12.599687    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 21:21:12.600129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:12.599971    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 21:21:16.725811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:16.725780    5886 generic.go:356] "Generic (PLEG): container finished" podID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerID="e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9" exitCode=0
Oct 30 21:21:16.726401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:16.725861    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" event={"ID":"2930b56b-c0fe-46a3-a059-04de254fb9f8","Type":"ContainerDied","Data":"e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9"}
Oct 30 21:21:16.726401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:16.726376    5886 scope.go:122] "RemoveContainer" containerID="e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9"
Oct 30 21:21:16.727784 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:16.727760    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:16.727852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:16.727826    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:16.727888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:16.727858    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:17.734667 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:17.734628    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" event={"ID":"2930b56b-c0fe-46a3-a059-04de254fb9f8","Type":"ContainerStarted","Data":"6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e"}
Oct 30 21:21:18.282567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:18.282535    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:21:18.282965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:18.282859    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:21:18.283175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:18.283134    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 21:21:19.281211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:19.280636    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:21:19.281211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:19.280710    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 21:21:20.279179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:20.278610    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:21:20.279179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:20.278673    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 21:21:21.278610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.278567    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:21:21.279995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.279959    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 21:21:21.448059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.448026    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"]
Oct 30 21:21:21.451669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.451647    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.471048 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.471004    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b31518-serving-cert\""
Oct 30 21:21:21.492018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.488667    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72dcd00-483a-42f3-90dd-14badea68753-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.492018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.488809    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.492018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.488864    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4tv\" (UniqueName: \"kubernetes.io/projected/d72dcd00-483a-42f3-90dd-14badea68753-kube-api-access-nh4tv\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.492018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.488962    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-metrics-certs\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.524294 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.524245    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"]
Oct 30 21:21:21.610725 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.610619    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.611017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.610978    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4tv\" (UniqueName: \"kubernetes.io/projected/d72dcd00-483a-42f3-90dd-14badea68753-kube-api-access-nh4tv\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.611295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.611277    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-metrics-certs\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.611485 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.611456    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72dcd00-483a-42f3-90dd-14badea68753-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.612844 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:21:21.612560    5886 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-serving-cert: secret "maas-api-e2e-worker-w1-b31518-serving-cert" not found
Oct 30 21:21:21.612844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.612589    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72dcd00-483a-42f3-90dd-14badea68753-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.612844 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:21:21.612638    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls podName:d72dcd00-483a-42f3-90dd-14badea68753 nodeName:}" failed. No retries permitted until 2026-10-30 21:21:22.112619016 +0000 UTC m=+682.241870464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls") pod "maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr" (UID: "d72dcd00-483a-42f3-90dd-14badea68753") : secret "maas-api-e2e-worker-w1-b31518-serving-cert" not found
Oct 30 21:21:21.615874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.615826    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-metrics-certs\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:21.631878 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:21.631856    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4tv\" (UniqueName: \"kubernetes.io/projected/d72dcd00-483a-42f3-90dd-14badea68753-kube-api-access-nh4tv\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:22.121077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:22.121041    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:22.125518 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:22.125491    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:22.281611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:22.281567    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:21:22.282574 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:22.282539    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 21:21:22.385102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:22.385004    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:22.788659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:22.788629    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"]
Oct 30 21:21:22.792958 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:21:22.792930    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72dcd00_483a_42f3_90dd_14badea68753.slice/crio-a14c1e3dfbbeb797af8530918f3323b6361c7bb93ee1a0fa98060cf57b9e3510 WatchSource:0}: Error finding container a14c1e3dfbbeb797af8530918f3323b6361c7bb93ee1a0fa98060cf57b9e3510: Status 404 returned error can't find the container with id a14c1e3dfbbeb797af8530918f3323b6361c7bb93ee1a0fa98060cf57b9e3510
Oct 30 21:21:23.283451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:23.283419    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:21:23.283957 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:23.283829    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:21:23.284721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:23.284695    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:21:23.774815 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:23.774780    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr" event={"ID":"d72dcd00-483a-42f3-90dd-14badea68753","Type":"ContainerStarted","Data":"63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca"}
Oct 30 21:21:23.775024 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:23.774838    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr" event={"ID":"d72dcd00-483a-42f3-90dd-14badea68753","Type":"ContainerStarted","Data":"a14c1e3dfbbeb797af8530918f3323b6361c7bb93ee1a0fa98060cf57b9e3510"}
Oct 30 21:21:23.813635 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:23.812915    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr" podStartSLOduration=2.424563453 podStartE2EDuration="2.812896867s" totalImagesPullingTime="388.333414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:23.810866521 +0000 UTC m=+683.940117977" watchObservedRunningTime="2026-10-30 21:21:23.812896867 +0000 UTC m=+683.942148325"
Oct 30 21:21:24.782345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:24.782307    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:26.795355 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:26.795322    5886 generic.go:356] "Generic (PLEG): container finished" podID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerID="a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68" exitCode=0
Oct 30 21:21:26.795770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:26.795395    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" event={"ID":"05864e5f-02e0-4bf2-b5b9-4130a34593dd","Type":"ContainerDied","Data":"a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68"}
Oct 30 21:21:26.795816 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:26.795775    5886 scope.go:122] "RemoveContainer" containerID="a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68"
Oct 30 21:21:26.796977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:26.796952    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:26.797025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:26.797015    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:26.797056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:26.797048    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:21:27.802590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:27.802553    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" event={"ID":"05864e5f-02e0-4bf2-b5b9-4130a34593dd","Type":"ContainerStarted","Data":"e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872"}
Oct 30 21:21:28.607940 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:28.607906    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:21:28.609079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:28.609047    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:28.609247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:28.609094    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 21:21:29.608771 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:29.608737    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:29.609214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:29.608790    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 21:21:30.608646 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:30.608600    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:30.608847 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:30.608653    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 21:21:30.795039 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:30.795009    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 21:21:31.608932 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:31.608894    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:31.609403 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:31.608951    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 21:21:32.596616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:32.596560    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:21:32.608200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:32.608172    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 21:21:32.608355 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:32.608219    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 21:21:33.613700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:33.613664    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:21:33.614965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:33.614939    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:21:41.028606 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:21:41.028566    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:21:58.112144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.112111    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"]
Oct 30 21:21:58.116302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.116279    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.133778 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.133748    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"]
Oct 30 21:21:58.143207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.140992    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"kube-root-ca.crt\""
Oct 30 21:21:58.143207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.141077    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"openshift-service-ca.crt\""
Oct 30 21:21:58.143207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.141123    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"default-dockercfg-8qqf5\""
Oct 30 21:21:58.143207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.141134    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-49738b\"/\"premium-w0-49738b-kserve-self-signed-certs\""
Oct 30 21:21:58.224089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.224052    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdln\" (UniqueName: \"kubernetes.io/projected/35fb6198-7dfd-44f0-a674-0d9287320862-kube-api-access-ctdln\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.224089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.224091    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-home\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.224306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.224118    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-model-cache\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.224306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.224262    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-dshm\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.224306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.224288    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-tmp-dir\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.224405 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.224369    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb6198-7dfd-44f0-a674-0d9287320862-tls-certs\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.326225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.326146    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb6198-7dfd-44f0-a674-0d9287320862-tls-certs\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.326560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.326536    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdln\" (UniqueName: \"kubernetes.io/projected/35fb6198-7dfd-44f0-a674-0d9287320862-kube-api-access-ctdln\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.326743 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.326724    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-home\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.326881 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.326867    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-model-cache\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.327052 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.327035    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-dshm\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.327200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.327175    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-tmp-dir\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.327558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.327534    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-home\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.327640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.327620    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-model-cache\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.327886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.327849    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-tmp-dir\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.329439 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.329418    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb6198-7dfd-44f0-a674-0d9287320862-tls-certs\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.329622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.329606    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-dshm\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.345094 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.345068    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdln\" (UniqueName: \"kubernetes.io/projected/35fb6198-7dfd-44f0-a674-0d9287320862-kube-api-access-ctdln\") pod \"premium-w0-49738b-kserve-6dc5bb4f59-g8hm5\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") " pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.429272 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.429235    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:21:58.733190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:58.733089    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"]
Oct 30 21:21:58.736876 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:21:58.736841    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35fb6198_7dfd_44f0_a674_0d9287320862.slice/crio-88035ef18dead5f4b287d9c8bb8d03331e682c7a76059c0291949858ddb4a829 WatchSource:0}: Error finding container 88035ef18dead5f4b287d9c8bb8d03331e682c7a76059c0291949858ddb4a829: Status 404 returned error can't find the container with id 88035ef18dead5f4b287d9c8bb8d03331e682c7a76059c0291949858ddb4a829
Oct 30 21:21:59.011004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:59.010909    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" event={"ID":"35fb6198-7dfd-44f0-a674-0d9287320862","Type":"ContainerStarted","Data":"3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328"}
Oct 30 21:21:59.011004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:59.010949    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" event={"ID":"35fb6198-7dfd-44f0-a674-0d9287320862","Type":"ContainerStarted","Data":"88035ef18dead5f4b287d9c8bb8d03331e682c7a76059c0291949858ddb4a829"}
Oct 30 21:21:59.049117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:59.049070    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podStartSLOduration=1.049057454 podStartE2EDuration="1.049057454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:59.048319511 +0000 UTC m=+719.177570968" watchObservedRunningTime="2026-10-30 21:21:59.049057454 +0000 UTC m=+719.178308911"
Oct 30 21:21:59.429741 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:21:59.429704    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:22:08.430082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:08.430038    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:22:08.443051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:08.443025    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:22:09.081598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:09.081566    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:22:11.028451 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:22:11.028362    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:22:23.626441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.626402    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"]
Oct 30 21:22:23.630840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.630796    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.646061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.646008    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ea40ee-openshift-default-dockercfg-8mf6h\""
Oct 30 21:22:23.660247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.660223    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"]
Oct 30 21:22:23.699180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699126    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc63398-277a-464d-928d-35c1d17164bb-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.699334 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699227    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7n5d\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-kube-api-access-n7n5d\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.699334 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699291    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.699334 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699330    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc63398-277a-464d-928d-35c1d17164bb-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.699442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699395    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.699442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699436    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.699505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699463    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.699540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699511    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.699572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.699562    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800215 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800182    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800229    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800341    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800489 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800409    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800556 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800518    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc63398-277a-464d-928d-35c1d17164bb-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800617 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800554    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7n5d\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-kube-api-access-n7n5d\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800617 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800605    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800744 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800615    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800744 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800660    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800744 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800668    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc63398-277a-464d-928d-35c1d17164bb-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800759    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.800897 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.800865    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.801055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.801036    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.801172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.801139    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc63398-277a-464d-928d-35c1d17164bb-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.802976 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.802952    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.803084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.803056    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc63398-277a-464d-928d-35c1d17164bb-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.818330 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.818308    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7n5d\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-kube-api-access-n7n5d\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.819119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.819100    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:23.945092 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:23.945056    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:24.265049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:24.264977    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"]
Oct 30 21:22:24.268695 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:22:24.268669    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc63398_277a_464d_928d_35c1d17164bb.slice/crio-7212899f41caa0fdfb39454793becb303ea791f6758d7f0320aadcf71cae5e79 WatchSource:0}: Error finding container 7212899f41caa0fdfb39454793becb303ea791f6758d7f0320aadcf71cae5e79: Status 404 returned error can't find the container with id 7212899f41caa0fdfb39454793becb303ea791f6758d7f0320aadcf71cae5e79
Oct 30 21:22:24.271450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:24.271418    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:22:24.271526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:24.271482    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:22:24.271526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:24.271510    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:22:25.193498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:25.193459    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" event={"ID":"fcc63398-277a-464d-928d-35c1d17164bb","Type":"ContainerStarted","Data":"643a11bbf99d93066ea208cdebe2ae32cc555756b1aa2be231c61e21330617a9"}
Oct 30 21:22:25.193931 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:25.193516    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" event={"ID":"fcc63398-277a-464d-928d-35c1d17164bb","Type":"ContainerStarted","Data":"7212899f41caa0fdfb39454793becb303ea791f6758d7f0320aadcf71cae5e79"}
Oct 30 21:22:25.945734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:25.945704    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:25.947117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:25.947094    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:25.947245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:25.947146    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 21:22:26.946251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:26.946212    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:26.946732 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:26.946281    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 21:22:27.945875 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:27.945839    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:27.946057 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:27.945903    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 21:22:28.946399 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:28.946367    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:28.946977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:28.946422    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 21:22:29.945880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:29.945837    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:22:29.946050 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:29.945916    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 21:22:30.949940 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:30.949909    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:30.950449 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:30.950257    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:30.951032 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:30.951007    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:22:31.007795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:22:31.007731    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" podStartSLOduration=8.007713437 podStartE2EDuration="8.007713437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:25.253927089 +0000 UTC m=+745.383178547" watchObservedRunningTime="2026-10-30 21:22:31.007713437 +0000 UTC m=+751.136964897"
Oct 30 21:22:41.031642 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:22:41.031598    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:23:09.795971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.795935    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"]
Oct 30 21:23:09.800974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.800947    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.820682 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.820656    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b31518-openshift-default-dockercfg-xdgb8\""
Oct 30 21:23:09.834318 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.834285    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"]
Oct 30 21:23:09.944112 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944070    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.944112 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944113    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.944345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944140    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-kube-api-access-b6k9m\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.944345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944261    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.944345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944288    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.944345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944307    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.944472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944432    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.944530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944517    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:09.944602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:09.944580    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.045515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045470    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.045726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045544    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.045726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045578    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.045726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045627    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.045726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045659    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.045726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045679    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-kube-api-access-b6k9m\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.045726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045716    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.046065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045940    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.046065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.045966    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.046065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.046005    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.046065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.046012    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.046372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.046347    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.046506 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.046400    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.046659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.046639    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.048555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.048530    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.048679 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.048587    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.067274 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.067240    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.067610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.067586    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-kube-api-access-b6k9m\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.115378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.115341    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:10.431577 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.431545    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"]
Oct 30 21:23:10.435196 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:23:10.435149    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652750b3_8ca9_481a_b675_1ee10b1b3a2e.slice/crio-09d49237ecb50b30d21484e1656c8fcb169c43f3517e0f1ba4cb32893283374b WatchSource:0}: Error finding container 09d49237ecb50b30d21484e1656c8fcb169c43f3517e0f1ba4cb32893283374b: Status 404 returned error can't find the container with id 09d49237ecb50b30d21484e1656c8fcb169c43f3517e0f1ba4cb32893283374b
Oct 30 21:23:10.437695 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.437662    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:10.437797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.437729    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:10.437797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.437759    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:10.474012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:10.473987    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" event={"ID":"652750b3-8ca9-481a-b675-1ee10b1b3a2e","Type":"ContainerStarted","Data":"09d49237ecb50b30d21484e1656c8fcb169c43f3517e0f1ba4cb32893283374b"}
Oct 30 21:23:11.028890 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:23:11.028855    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:23:11.498370 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:11.498333    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" event={"ID":"652750b3-8ca9-481a-b675-1ee10b1b3a2e","Type":"ContainerStarted","Data":"eea3e934e803696f2b9c252c28eee4c3d1e989b13773aff45624a926138f71d4"}
Oct 30 21:23:11.551627 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:11.551572    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podStartSLOduration=2.551558129 podStartE2EDuration="2.551558129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:11.549349607 +0000 UTC m=+791.678601065" watchObservedRunningTime="2026-10-30 21:23:11.551558129 +0000 UTC m=+791.680809585"
Oct 30 21:23:12.115623 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:12.115586    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:12.116992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:12.116964    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:23:12.117145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:12.117014    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:23:13.116503 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:13.115912    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:23:13.116503 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:13.115998    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:23:14.116336 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:14.116297    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:23:14.116506 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:14.116375    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:23:15.115825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:15.115795    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:23:15.116325 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:15.116297    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:23:16.117178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:16.116601    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:23:16.117178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:16.116727    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:23:17.116126 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:17.116076    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:23:17.116367 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:17.116150    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:23:18.121017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:18.120987    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:18.121536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:18.121373    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:18.122042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:18.122022    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:23:24.428173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.428127    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"]
Oct 30 21:23:24.433247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.433218    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.475989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.475944    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"]
Oct 30 21:23:24.511032 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511000    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.511214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511144    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.511299 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511219    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e24781-6814-4429-9f0f-c47017248f06-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.511349 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511315    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.511451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511430    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.511517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511502    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e24781-6814-4429-9f0f-c47017248f06-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.511569 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511554    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffw5\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-kube-api-access-5ffw5\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.511660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511641    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.511717 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.511697    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.612432 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612389    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.612432 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612439    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.612720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612500    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.612720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612578    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.612720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612606    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e24781-6814-4429-9f0f-c47017248f06-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.612720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612631    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.612720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612694    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.613027 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612750    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e24781-6814-4429-9f0f-c47017248f06-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.613027 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.612796    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffw5\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-kube-api-access-5ffw5\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.613514 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.613487    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.614205 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.614184    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.614467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.614437    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.615698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.615678    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.616266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.616209    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e24781-6814-4429-9f0f-c47017248f06-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.616266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.616230    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.618803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.618777    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e24781-6814-4429-9f0f-c47017248f06-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.631858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.631651    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffw5\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-kube-api-access-5ffw5\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.633392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.633367    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:24.756573 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:24.756462    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:25.149585 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:25.147316    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"]
Oct 30 21:23:25.159875 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:25.159831    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:25.159993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:25.159920    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:25.159993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:25.159968    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:23:25.660199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:25.660050    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" event={"ID":"d6e24781-6814-4429-9f0f-c47017248f06","Type":"ContainerStarted","Data":"6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6"}
Oct 30 21:23:25.660199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:25.660102    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" event={"ID":"d6e24781-6814-4429-9f0f-c47017248f06","Type":"ContainerStarted","Data":"280d2c6462accedf9ee9d9a38e642496fc2b15d851c1124675e75f34f9551300"}
Oct 30 21:23:25.707179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:25.707057    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podStartSLOduration=1.707037624 podStartE2EDuration="1.707037624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:25.703630659 +0000 UTC m=+805.832882119" watchObservedRunningTime="2026-10-30 21:23:25.707037624 +0000 UTC m=+805.836289082"
Oct 30 21:23:25.758176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:25.757413    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:26.775197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:26.774868    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:23:26.775197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:26.774924    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:23:27.757576 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:27.757540    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:23:27.757765 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:27.757591    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:23:28.757681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:28.757647    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:23:28.758058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:28.757701    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:23:29.757274 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:29.757237    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:23:29.757457 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:29.757322    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:23:30.757706 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:30.757669    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:23:30.758117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:30.757725    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:23:31.757271 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:31.757238    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:23:31.757481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:31.757299    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:23:32.929861 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:32.929832    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:32.933254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:32.932047    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:32.933254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:32.933088    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:23:40.676506 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.676459    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"]
Oct 30 21:23:40.681437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.681413    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.695698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.695623    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"openshift-service-ca.crt\""
Oct 30 21:23:40.711443 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.711384    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"]
Oct 30 21:23:40.711574 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.711502    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"kube-root-ca.crt\""
Oct 30 21:23:40.711631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.711597    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"default-dockercfg-ktmlc\""
Oct 30 21:23:40.711691 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.711674    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"distinct-w3-50ab77-kserve-self-signed-certs\""
Oct 30 21:23:40.765101 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.765073    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-tmp-dir\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.765266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.765117    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1020a874-7ec4-44be-905c-792bc7f714e1-tls-certs\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.765266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.765234    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-dshm\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.765398 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.765358    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-home\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.765501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.765452    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-model-cache\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.765547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.765512    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcpr\" (UniqueName: \"kubernetes.io/projected/1020a874-7ec4-44be-905c-792bc7f714e1-kube-api-access-zkcpr\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867138 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867102    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-model-cache\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867147    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcpr\" (UniqueName: \"kubernetes.io/projected/1020a874-7ec4-44be-905c-792bc7f714e1-kube-api-access-zkcpr\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867217    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-tmp-dir\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867350    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1020a874-7ec4-44be-905c-792bc7f714e1-tls-certs\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867410    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-dshm\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867483    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-home\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867539    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-model-cache\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867552    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-tmp-dir\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.867817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.867796    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-home\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.870052 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.870030    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1020a874-7ec4-44be-905c-792bc7f714e1-tls-certs\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.889147 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.888466    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-dshm\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:40.896679 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:40.896645    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcpr\" (UniqueName: \"kubernetes.io/projected/1020a874-7ec4-44be-905c-792bc7f714e1-kube-api-access-zkcpr\") pod \"distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:41.018537 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.018455    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:41.031527 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:23:41.031467    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:23:41.326905 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.326806    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"]
Oct 30 21:23:41.331371 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:23:41.331339    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1020a874_7ec4_44be_905c_792bc7f714e1.slice/crio-681fcb69096aa040e27b70711494f6ae7b96c6951383f715ae7fac026b2d4488 WatchSource:0}: Error finding container 681fcb69096aa040e27b70711494f6ae7b96c6951383f715ae7fac026b2d4488: Status 404 returned error can't find the container with id 681fcb69096aa040e27b70711494f6ae7b96c6951383f715ae7fac026b2d4488
Oct 30 21:23:41.801234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.801197    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" event={"ID":"1020a874-7ec4-44be-905c-792bc7f714e1","Type":"ContainerStarted","Data":"0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f"}
Oct 30 21:23:41.801735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.801244    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" event={"ID":"1020a874-7ec4-44be-905c-792bc7f714e1","Type":"ContainerStarted","Data":"681fcb69096aa040e27b70711494f6ae7b96c6951383f715ae7fac026b2d4488"}
Oct 30 21:23:41.836995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.836944    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podStartSLOduration=1.836932067 podStartE2EDuration="1.836932067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:41.836458567 +0000 UTC m=+821.965710025" watchObservedRunningTime="2026-10-30 21:23:41.836932067 +0000 UTC m=+821.966183524"
Oct 30 21:23:41.963401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.963357    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"]
Oct 30 21:23:41.967892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.967867    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:41.981983 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.981953    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-routing-overlay-volume\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:41.982125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.982029    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-plugins-config-volume\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:41.982243 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.982122    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-provider-credentials\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:41.984104 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.984076    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dmwj5\""
Oct 30 21:23:41.984270 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.984200    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:23:41.984270 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.984260    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:23:41.984386 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.984315    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-18092bbdb3eb\""
Oct 30 21:23:41.984439 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.984389    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:23:41.984485 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.984441    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:23:41.988125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.986712    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6q9f\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-kube-api-access-m6q9f\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:41.997858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:41.997829    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"]
Oct 30 21:23:42.019179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.019137    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:42.087401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.087321    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-provider-credentials\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.087575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.087421    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6q9f\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-kube-api-access-m6q9f\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.087575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.087458    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-routing-overlay-volume\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.087706 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.087666    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-plugins-config-volume\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.088303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.088275    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-routing-overlay-volume\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.088443 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.088397    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-plugins-config-volume\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.089969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.089947    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-provider-credentials\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.106691 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.106666    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6q9f\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-kube-api-access-m6q9f\") pod \"payload-processing-external-model-7b8868c9c7-xfdrb\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") " pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.305376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.305339    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:42.610945 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.610847    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"]
Oct 30 21:23:42.616745 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:23:42.616712    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2916c977_34fa_4771_843d_1b999247b638.slice/crio-dfa8d81ea61f9479615d46b72ef9cbb2ff706b559d3ff57c036ba6c26273bd0b WatchSource:0}: Error finding container dfa8d81ea61f9479615d46b72ef9cbb2ff706b559d3ff57c036ba6c26273bd0b: Status 404 returned error can't find the container with id dfa8d81ea61f9479615d46b72ef9cbb2ff706b559d3ff57c036ba6c26273bd0b
Oct 30 21:23:42.818659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.818610    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" event={"ID":"2916c977-34fa-4771-843d-1b999247b638","Type":"ContainerStarted","Data":"b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb"}
Oct 30 21:23:42.819067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.818703    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" event={"ID":"2916c977-34fa-4771-843d-1b999247b638","Type":"ContainerStarted","Data":"dfa8d81ea61f9479615d46b72ef9cbb2ff706b559d3ff57c036ba6c26273bd0b"}
Oct 30 21:23:42.856635 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:42.856588    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podStartSLOduration=1.856575543 podStartE2EDuration="1.856575543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:42.855737037 +0000 UTC m=+822.984988494" watchObservedRunningTime="2026-10-30 21:23:42.856575543 +0000 UTC m=+822.985827018"
Oct 30 21:23:43.824285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:43.824252    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:23:46.537712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.537678    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"]
Oct 30 21:23:46.542087 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.542051    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.560660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.560628    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"]
Oct 30 21:23:46.567618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.567589    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"kube-root-ca.crt\""
Oct 30 21:23:46.567618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.567588    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"openshift-service-ca.crt\""
Oct 30 21:23:46.567848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.567593    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"default-dockercfg-vrsxp\""
Oct 30 21:23:46.567848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.567687    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"e2e-conflict-b-26510fbf-kserve-self-signed-certs\""
Oct 30 21:23:46.655115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.655083    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-model-cache\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.655315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.655129    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkqf\" (UniqueName: \"kubernetes.io/projected/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-kube-api-access-8rkqf\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.655315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.655231    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tls-certs\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.655398 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.655368    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-home\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.655442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.655400    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-dshm\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.655540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.655524    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tmp-dir\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.763060 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.763017    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-model-cache\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.763525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.763502    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkqf\" (UniqueName: \"kubernetes.io/projected/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-kube-api-access-8rkqf\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.764016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.763616    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-model-cache\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.764584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.764564    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tls-certs\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.764794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.764777    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-home\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.764970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.764954    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-dshm\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.765213 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.765192    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tmp-dir\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.765776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.765756    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tmp-dir\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.766214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.766186    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-home\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.769958 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.769904    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-dshm\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.771422 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.771402    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tls-certs\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.782946 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.782910    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkqf\" (UniqueName: \"kubernetes.io/projected/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-kube-api-access-8rkqf\") pod \"e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") " pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:46.859776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:46.859670    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:47.168575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:47.168538    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"]
Oct 30 21:23:47.173838 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:23:47.173798    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8a88beb_5f1c_4a46_b6b3_55fef7278cca.slice/crio-c0bc3d65b5678a9a20e1214b577030fba95c1d09611fd579c4d4d0bfb88708ae WatchSource:0}: Error finding container c0bc3d65b5678a9a20e1214b577030fba95c1d09611fd579c4d4d0bfb88708ae: Status 404 returned error can't find the container with id c0bc3d65b5678a9a20e1214b577030fba95c1d09611fd579c4d4d0bfb88708ae
Oct 30 21:23:47.851502 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:47.851457    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8" event={"ID":"f8a88beb-5f1c-4a46-b6b3-55fef7278cca","Type":"ContainerStarted","Data":"2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565"}
Oct 30 21:23:47.851963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:47.851515    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8" event={"ID":"f8a88beb-5f1c-4a46-b6b3-55fef7278cca","Type":"ContainerStarted","Data":"c0bc3d65b5678a9a20e1214b577030fba95c1d09611fd579c4d4d0bfb88708ae"}
Oct 30 21:23:47.860183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:47.860149    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:47.891273 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:47.891210    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8" podStartSLOduration=1.891195129 podStartE2EDuration="1.891195129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:47.888320241 +0000 UTC m=+828.017571697" watchObservedRunningTime="2026-10-30 21:23:47.891195129 +0000 UTC m=+828.020446585"
Oct 30 21:23:51.019119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:51.019078    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:51.032141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:51.032107    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:51.891740 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:51.891706    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:23:54.839472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:54.839053    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:23:56.859869 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:56.859825    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:56.873057 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:56.873031    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:23:56.958125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:23:56.958093    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:24:04.832350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:04.832312    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:08.945437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:08.943820    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"]
Oct 30 21:24:08.950006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:08.949977    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:08.963309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:08.963274    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"]
Oct 30 21:24:08.969413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:08.969385    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:08.974855 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:08.974831    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"]
Oct 30 21:24:08.989973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:08.989946    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"]
Oct 30 21:24:09.114968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.114931    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b822225a-462c-4fdf-af75-c411a958b44f-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.114979    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdbhk\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-kube-api-access-zdbhk\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.115169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115076    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.115169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115103    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.115169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115135    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.115336 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115227    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115372    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115445 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115430    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115486 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115470    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.115519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115502    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115582 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115564    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.115639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115628    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.115675 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115659    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqtw\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-kube-api-access-gmqtw\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115715 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115683    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115758 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115725    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.115799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115758    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b822225a-462c-4fdf-af75-c411a958b44f-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115801    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.115837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.115821    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.216920    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.216959    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.216985    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217002    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217034    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217062    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217079    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217294    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217338    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217423    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217439    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217455    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqtw\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-kube-api-access-gmqtw\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217473    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217499    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217560    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217584    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217645    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217759    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b822225a-462c-4fdf-af75-c411a958b44f-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217780    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217796    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217817    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b822225a-462c-4fdf-af75-c411a958b44f-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217836    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdbhk\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-kube-api-access-zdbhk\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.217965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217845    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.217965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.217868    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.218281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.218066    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.218281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.218232    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.218739 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.218714    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b822225a-462c-4fdf-af75-c411a958b44f-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.218874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.218767    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.219861 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.219837    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.220253 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.220224    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b822225a-462c-4fdf-af75-c411a958b44f-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.220349 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.220320    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.220953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.220933    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.235566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.235535    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.235719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.235700    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdbhk\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-kube-api-access-zdbhk\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.235787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.235734    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqtw\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-kube-api-access-gmqtw\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.235874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.235858    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.273730 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.273697    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:09.294016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.293771    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:09.617324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.617246    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"]
Oct 30 21:24:09.620925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.620897    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"]
Oct 30 21:24:09.625221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.625190    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:09.625338 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.625268    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:09.625338 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.625315    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:09.625956 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:24:09.625930    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda668566_a5aa_4f42_a0b6_02c6829bcbe9.slice/crio-0ddb78bf9c2044cf02e32ca95ecc9227e731beada861d5be62fcd1fa091b45c4 WatchSource:0}: Error finding container 0ddb78bf9c2044cf02e32ca95ecc9227e731beada861d5be62fcd1fa091b45c4: Status 404 returned error can't find the container with id 0ddb78bf9c2044cf02e32ca95ecc9227e731beada861d5be62fcd1fa091b45c4
Oct 30 21:24:09.628002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.627974    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:09.628066 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.628048    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:09.628114 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:09.628099    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:10.056981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.056942    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" event={"ID":"da668566-a5aa-4f42-a0b6-02c6829bcbe9","Type":"ContainerStarted","Data":"b3929457a920d813c5986087872801a4c96a3d4bafe9cd2af7d612508468c4e6"}
Oct 30 21:24:10.057488 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.057004    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" event={"ID":"da668566-a5aa-4f42-a0b6-02c6829bcbe9","Type":"ContainerStarted","Data":"0ddb78bf9c2044cf02e32ca95ecc9227e731beada861d5be62fcd1fa091b45c4"}
Oct 30 21:24:10.058732 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.058701    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" event={"ID":"b822225a-462c-4fdf-af75-c411a958b44f","Type":"ContainerStarted","Data":"40d1c4a907c618d7a4d213ced0e44d2aa56104315c0a17897756e6de4a584a9f"}
Oct 30 21:24:10.058881 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.058751    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" event={"ID":"b822225a-462c-4fdf-af75-c411a958b44f","Type":"ContainerStarted","Data":"8709797fdfdea5f2ad892c367e0608519d34b3c55d37a55dfc2bce14dbe6b258"}
Oct 30 21:24:10.099562 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.099488    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podStartSLOduration=2.099468494 podStartE2EDuration="2.099468494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:10.097454889 +0000 UTC m=+850.226706345" watchObservedRunningTime="2026-10-30 21:24:10.099468494 +0000 UTC m=+850.228719955"
Oct 30 21:24:10.142647 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.142575    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podStartSLOduration=2.142555852 podStartE2EDuration="2.142555852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:10.139338248 +0000 UTC m=+850.268589706" watchObservedRunningTime="2026-10-30 21:24:10.142555852 +0000 UTC m=+850.271807310"
Oct 30 21:24:10.274280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.274247    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:10.275776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.275743    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:24:10.275912 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.275789    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:24:10.294293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.294260    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:10.295922 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.295892    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:24:10.296041 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:10.295944    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:24:11.046323 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:24:11.046289    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:24:11.283177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:11.279558    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:24:11.283177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:11.279634    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:24:11.302239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:11.298192    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:24:11.302239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:11.298252    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:24:12.282180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:12.281558    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:24:12.282180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:12.281631    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:24:12.296563 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:12.296528    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:24:12.297148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:12.297117    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:24:13.276610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:13.276566    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:24:13.276900 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:13.276871    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:24:13.300175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:13.300123    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:24:13.300772 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:13.300743    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:24:14.277624 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:14.277582    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:24:14.277969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:14.277940    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:24:14.296229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:14.296197    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:24:14.296525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:14.296462    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:24:14.836269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:14.836223    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:15.285208 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:15.282513    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:24:15.285208 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:15.282578    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:24:15.310179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:15.302960    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:24:15.310179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:15.303019    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:24:16.274531 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:16.274492    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:24:16.275040 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:16.274554    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:24:16.295290 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:16.295247    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:24:16.295478 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:16.295321    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:24:17.479540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:17.479500    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:17.483656 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:17.482223    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:17.483656 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:17.482314    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:17.483656 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:17.483130    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:17.484102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:17.484080    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:24:17.485668 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:17.485612    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:24:20.734443 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:20.734403    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"]
Oct 30 21:24:23.594862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.594822    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"]
Oct 30 21:24:23.599915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.599890    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.626656 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.626601    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-50ab77\"/\"distinct-2-w3-50ab77-kserve-self-signed-certs\""
Oct 30 21:24:23.629453 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.629423    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"]
Oct 30 21:24:23.775081 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.775044    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqcj\" (UniqueName: \"kubernetes.io/projected/199b2e2a-8baa-47fd-9361-a3018b22fc4f-kube-api-access-5tqcj\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.775281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.775107    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-home\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.775281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.775128    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-dshm\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.775281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.775276    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-model-cache\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.775471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.775366    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tmp-dir\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.775471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.775414    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tls-certs\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.876952    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tmp-dir\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.877028    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tls-certs\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.877099    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqcj\" (UniqueName: \"kubernetes.io/projected/199b2e2a-8baa-47fd-9361-a3018b22fc4f-kube-api-access-5tqcj\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.877199    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-home\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.877228    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-dshm\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877462 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.877339    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-model-cache\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877462 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.877422    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tmp-dir\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877553 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.877521    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-home\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.877983 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.877959    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-model-cache\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.879587 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.879561    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tls-certs\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.879899 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.879878    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-dshm\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.895320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.895294    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqcj\" (UniqueName: \"kubernetes.io/projected/199b2e2a-8baa-47fd-9361-a3018b22fc4f-kube-api-access-5tqcj\") pod \"distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") " pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:23.917815 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:23.917784    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:24.232235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:24.232200    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"]
Oct 30 21:24:24.236368 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:24:24.236336    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod199b2e2a_8baa_47fd_9361_a3018b22fc4f.slice/crio-5aaf85b3a6b90aea78cd35914db65fd11a0a98980924e5f796341cf616252b9b WatchSource:0}: Error finding container 5aaf85b3a6b90aea78cd35914db65fd11a0a98980924e5f796341cf616252b9b: Status 404 returned error can't find the container with id 5aaf85b3a6b90aea78cd35914db65fd11a0a98980924e5f796341cf616252b9b
Oct 30 21:24:24.832175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:24.832119    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:25.050461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.050411    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"]
Oct 30 21:24:25.055862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.055836    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.075815 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.075783    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"]
Oct 30 21:24:25.195551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.195505    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.195551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.195544    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.195819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.195575    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9spz\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-kube-api-access-k9spz\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.195819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.195693    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.195819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.195736    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.195819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.195756    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.196042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.195830    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.196042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.195909    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.196174 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.196113    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.246133 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.246098    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" event={"ID":"199b2e2a-8baa-47fd-9361-a3018b22fc4f","Type":"ContainerStarted","Data":"b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f"}
Oct 30 21:24:25.246320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.246141    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" event={"ID":"199b2e2a-8baa-47fd-9361-a3018b22fc4f","Type":"ContainerStarted","Data":"5aaf85b3a6b90aea78cd35914db65fd11a0a98980924e5f796341cf616252b9b"}
Oct 30 21:24:25.283703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.283640    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podStartSLOduration=2.283625761 podStartE2EDuration="2.283625761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:25.282487869 +0000 UTC m=+865.411739327" watchObservedRunningTime="2026-10-30 21:24:25.283625761 +0000 UTC m=+865.412877276"
Oct 30 21:24:25.297278 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.297243    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.297451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.297295    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.297451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.297433    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.297553 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.297477    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.297553 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.297540    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.297757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.297734    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.298097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.297993    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.298274 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.298254    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.298414 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.298395    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9spz\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-kube-api-access-k9spz\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.298561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.297784    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.298641 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.298561    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.298641 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.298014    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.298641 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.298612    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.298798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.298278    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.300088 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.300063    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.300297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.300275    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.317395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.317367    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.317681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.317664    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9spz\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-kube-api-access-k9spz\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.372412 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.372378    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:25.714981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.714906    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"]
Oct 30 21:24:25.718316 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:24:25.718289    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59caa8f3_3d24_4f7a_8d4f_78914d2cef09.slice/crio-a655f3e8bbfe9000994b53edd1ef7c6e7dfb07213c3494d9edcaa51ed88d61c8 WatchSource:0}: Error finding container a655f3e8bbfe9000994b53edd1ef7c6e7dfb07213c3494d9edcaa51ed88d61c8: Status 404 returned error can't find the container with id a655f3e8bbfe9000994b53edd1ef7c6e7dfb07213c3494d9edcaa51ed88d61c8
Oct 30 21:24:25.720808 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.720774    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:25.720909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.720838    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:25.720909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.720868    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:25.918347 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:25.918291    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:26.258148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:26.258099    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" event={"ID":"59caa8f3-3d24-4f7a-8d4f-78914d2cef09","Type":"ContainerStarted","Data":"351ff491b0b6149e0634c16e59a92aa4545ed1d52701b52105d19de896018759"}
Oct 30 21:24:26.258372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:26.258209    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" event={"ID":"59caa8f3-3d24-4f7a-8d4f-78914d2cef09","Type":"ContainerStarted","Data":"a655f3e8bbfe9000994b53edd1ef7c6e7dfb07213c3494d9edcaa51ed88d61c8"}
Oct 30 21:24:26.399179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:26.398527    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:26.400363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:26.400304    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podStartSLOduration=1.400284699 podStartE2EDuration="1.400284699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:26.328611254 +0000 UTC m=+866.457862726" watchObservedRunningTime="2026-10-30 21:24:26.400284699 +0000 UTC m=+866.529536156"
Oct 30 21:24:26.410345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:26.410308    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:24:26.414315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:26.414277    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:24:27.386181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:27.385301    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:24:27.386181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:27.385370    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:24:28.394204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:28.386708    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:24:28.394204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:28.386766    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:24:29.373603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:29.373562    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:24:29.373910 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:29.373880    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:24:30.384966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:30.384535    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:24:30.384966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:30.384600    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:24:31.376593 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:31.376546    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:24:31.376787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:31.376614    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:24:32.373651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:32.373617    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:24:32.374178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:32.373668    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:24:33.378189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:33.378131    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:33.378623 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:33.378567    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:33.379265 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:33.379245    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:24:33.918670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:33.918638    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:33.932384 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:33.932355    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:34.357271 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:34.357179    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:34.831756 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:34.831720    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:35.755856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:35.755817    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8" podUID="f8a88beb-5f1c-4a46-b6b3-55fef7278cca" containerName="main" containerID="cri-o://2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565" gracePeriod=45
Oct 30 21:24:36.027113 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.027084    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:24:36.199825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.199788    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tls-certs\") pod \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") "
Oct 30 21:24:36.200023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.199837    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-dshm\") pod \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") "
Oct 30 21:24:36.200023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.199916    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-kube-api-access-8rkqf\" (UniqueName: \"kubernetes.io/projected/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-kube-api-access-8rkqf\") pod \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") "
Oct 30 21:24:36.200023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.199946    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-model-cache\") pod \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") "
Oct 30 21:24:36.200251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.200089    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tmp-dir\") pod \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") "
Oct 30 21:24:36.200251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.200132    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-home\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-home\") pod \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\" (UID: \"f8a88beb-5f1c-4a46-b6b3-55fef7278cca\") "
Oct 30 21:24:36.200820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.200474    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-model-cache" pod "f8a88beb-5f1c-4a46-b6b3-55fef7278cca" (UID: "f8a88beb-5f1c-4a46-b6b3-55fef7278cca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:36.200820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.200545    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tmp-dir" pod "f8a88beb-5f1c-4a46-b6b3-55fef7278cca" (UID: "f8a88beb-5f1c-4a46-b6b3-55fef7278cca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:36.200820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.200766    5886 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:36.200820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.200792    5886 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:36.201045 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.200836    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-home" pod "f8a88beb-5f1c-4a46-b6b3-55fef7278cca" (UID: "f8a88beb-5f1c-4a46-b6b3-55fef7278cca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:36.202794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.202757    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-dshm" pod "f8a88beb-5f1c-4a46-b6b3-55fef7278cca" (UID: "f8a88beb-5f1c-4a46-b6b3-55fef7278cca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:36.202794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.202769    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tls-certs" pod "f8a88beb-5f1c-4a46-b6b3-55fef7278cca" (UID: "f8a88beb-5f1c-4a46-b6b3-55fef7278cca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:36.203002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.202837    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-kube-api-access-8rkqf" pod "f8a88beb-5f1c-4a46-b6b3-55fef7278cca" (UID: "f8a88beb-5f1c-4a46-b6b3-55fef7278cca"). InnerVolumeSpecName "kube-api-access-8rkqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:36.302463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.302374    5886 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:36.302463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.302411    5886 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:36.302463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.302426    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rkqf\" (UniqueName: \"kubernetes.io/projected/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-kube-api-access-8rkqf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:36.302463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.302438    5886 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a88beb-5f1c-4a46-b6b3-55fef7278cca-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:36.360276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.360244    5886 generic.go:356] "Generic (PLEG): container finished" podID="f8a88beb-5f1c-4a46-b6b3-55fef7278cca" containerID="2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565" exitCode=0
Oct 30 21:24:36.360456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.360296    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"
Oct 30 21:24:36.360456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.360305    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8" event={"ID":"f8a88beb-5f1c-4a46-b6b3-55fef7278cca","Type":"ContainerDied","Data":"2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565"}
Oct 30 21:24:36.360456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.360349    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8" event={"ID":"f8a88beb-5f1c-4a46-b6b3-55fef7278cca","Type":"ContainerDied","Data":"c0bc3d65b5678a9a20e1214b577030fba95c1d09611fd579c4d4d0bfb88708ae"}
Oct 30 21:24:36.360456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.360366    5886 scope.go:122] "RemoveContainer" containerID="2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565"
Oct 30 21:24:36.372704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.372683    5886 scope.go:122] "RemoveContainer" containerID="2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565"
Oct 30 21:24:36.372989 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:24:36.372970    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565\": container with ID starting with 2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565 not found: ID does not exist" containerID="2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565"
Oct 30 21:24:36.373047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.372997    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565"} err="failed to get container status \"2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565\": rpc error: code = NotFound desc = could not find container \"2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565\": container with ID starting with 2f9c04a26e48edf0f6abb510a6f721433763cea2fd13d27980cd16821088e565 not found: ID does not exist"
Oct 30 21:24:36.409654 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.409620    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"]
Oct 30 21:24:36.419763 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.419738    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8"]
Oct 30 21:24:36.948259 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:36.948211    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/e2e-conflict-b-26510fbf-kserve-5bff5cb885-jghm8" podUID="f8a88beb-5f1c-4a46-b6b3-55fef7278cca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:38.009694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:38.009275    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a88beb-5f1c-4a46-b6b3-55fef7278cca" path="/var/lib/kubelet/pods/f8a88beb-5f1c-4a46-b6b3-55fef7278cca/volumes"
Oct 30 21:24:41.032089 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:24:41.032043    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:24:43.932597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.932562    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"]
Oct 30 21:24:43.933371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.933342    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f8a88beb-5f1c-4a46-b6b3-55fef7278cca" containerName="main"
Oct 30 21:24:43.933371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.933368    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a88beb-5f1c-4a46-b6b3-55fef7278cca" containerName="main"
Oct 30 21:24:43.933569 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.933545    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a88beb-5f1c-4a46-b6b3-55fef7278cca" containerName="main"
Oct 30 21:24:43.938397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.938371    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:43.965081 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.965053    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"openshift-service-ca.crt\""
Oct 30 21:24:43.966366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.966119    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"default-dockercfg-vrsxp\""
Oct 30 21:24:43.966741 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.966235    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"kube-root-ca.crt\""
Oct 30 21:24:43.966854 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.966307    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ea40ee\"/\"distinct-w4-ea40ee-kserve-self-signed-certs\""
Oct 30 21:24:43.970967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:43.970942    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"]
Oct 30 21:24:44.011396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.011361    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-dshm\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.011560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.011481    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-model-cache\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.011560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.011513    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlgtz\" (UniqueName: \"kubernetes.io/projected/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-kube-api-access-tlgtz\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.011698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.011576    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-home\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.011698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.011635    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tmp-dir\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.011698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.011653    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tls-certs\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.112378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.112347    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tmp-dir\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.112378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.112379    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tls-certs\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.112607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.112465    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-dshm\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.112607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.112546    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-model-cache\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.112720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.112690    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlgtz\" (UniqueName: \"kubernetes.io/projected/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-kube-api-access-tlgtz\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.112780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.112758    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-home\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.112896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.112785    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tmp-dir\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.112896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.112840    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-model-cache\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.113072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.113049    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-home\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.115000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.114979    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-dshm\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.115450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.115430    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tls-certs\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.132771 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.132738    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlgtz\" (UniqueName: \"kubernetes.io/projected/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-kube-api-access-tlgtz\") pod \"distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") " pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.254807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.254719    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:44.568873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.568790    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"]
Oct 30 21:24:44.573142 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:24:44.573115    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5c42ccf_fbaa_4ec6_932c_1cc63c44b0b0.slice/crio-184886ef842f3f0475f83fa084a924f2fc600b85774ba9c07bf920921547ed21 WatchSource:0}: Error finding container 184886ef842f3f0475f83fa084a924f2fc600b85774ba9c07bf920921547ed21: Status 404 returned error can't find the container with id 184886ef842f3f0475f83fa084a924f2fc600b85774ba9c07bf920921547ed21
Oct 30 21:24:44.832278 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:44.832188    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:45.458762 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:45.458726    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" event={"ID":"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0","Type":"ContainerStarted","Data":"b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764"}
Oct 30 21:24:45.458762 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:45.458774    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" event={"ID":"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0","Type":"ContainerStarted","Data":"184886ef842f3f0475f83fa084a924f2fc600b85774ba9c07bf920921547ed21"}
Oct 30 21:24:45.497460 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:45.497410    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podStartSLOduration=2.497395292 podStartE2EDuration="2.497395292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:45.495480118 +0000 UTC m=+885.624731575" watchObservedRunningTime="2026-10-30 21:24:45.497395292 +0000 UTC m=+885.626646751"
Oct 30 21:24:46.255102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:46.255068    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:47.731302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:47.731264    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"]
Oct 30 21:24:47.732709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:47.732678    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:52.732304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:52.732259    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing" containerID="cri-o://b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb" gracePeriod=25
Oct 30 21:24:53.028785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.028761    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:24:53.124339 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.124304    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-kube-api-access-m6q9f\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-kube-api-access-m6q9f\") pod \"2916c977-34fa-4771-843d-1b999247b638\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") "
Oct 30 21:24:53.124539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.124374    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-provider-credentials\") pod \"2916c977-34fa-4771-843d-1b999247b638\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") "
Oct 30 21:24:53.124539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.124410    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-routing-overlay-volume\") pod \"2916c977-34fa-4771-843d-1b999247b638\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") "
Oct 30 21:24:53.124672 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.124635    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-plugins-config-volume\") pod \"2916c977-34fa-4771-843d-1b999247b638\" (UID: \"2916c977-34fa-4771-843d-1b999247b638\") "
Oct 30 21:24:53.124896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.124865    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-routing-overlay-volume" pod "2916c977-34fa-4771-843d-1b999247b638" (UID: "2916c977-34fa-4771-843d-1b999247b638"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:53.125027 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.125006    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-plugins-config-volume" pod "2916c977-34fa-4771-843d-1b999247b638" (UID: "2916c977-34fa-4771-843d-1b999247b638"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:53.125214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.125193    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:53.125304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.125216    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2916c977-34fa-4771-843d-1b999247b638-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:53.126721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.126696    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-provider-credentials" pod "2916c977-34fa-4771-843d-1b999247b638" (UID: "2916c977-34fa-4771-843d-1b999247b638"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:53.126828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.126721    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-kube-api-access-m6q9f" pod "2916c977-34fa-4771-843d-1b999247b638" (UID: "2916c977-34fa-4771-843d-1b999247b638"). InnerVolumeSpecName "kube-api-access-m6q9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:53.226710 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.226640    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6q9f\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-kube-api-access-m6q9f\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:53.226710 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.226680    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2916c977-34fa-4771-843d-1b999247b638-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:24:53.517002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.516965    5886 generic.go:356] "Generic (PLEG): container finished" podID="2916c977-34fa-4771-843d-1b999247b638" containerID="b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb" exitCode=0
Oct 30 21:24:53.517237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.517016    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"
Oct 30 21:24:53.517237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.517029    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" event={"ID":"2916c977-34fa-4771-843d-1b999247b638","Type":"ContainerDied","Data":"b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb"}
Oct 30 21:24:53.517237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.517070    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb" event={"ID":"2916c977-34fa-4771-843d-1b999247b638","Type":"ContainerDied","Data":"dfa8d81ea61f9479615d46b72ef9cbb2ff706b559d3ff57c036ba6c26273bd0b"}
Oct 30 21:24:53.517237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.517093    5886 scope.go:122] "RemoveContainer" containerID="b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb"
Oct 30 21:24:53.527254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.527234    5886 scope.go:122] "RemoveContainer" containerID="b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb"
Oct 30 21:24:53.527513 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:24:53.527493    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb\": container with ID starting with b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb not found: ID does not exist" containerID="b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb"
Oct 30 21:24:53.527584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.527523    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb"} err="failed to get container status \"b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb\": rpc error: code = NotFound desc = could not find container \"b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb\": container with ID starting with b3c326dbb5aae049dbc838a85d914b6e4c2fd11650eb03420f12aa36f7fb7bcb not found: ID does not exist"
Oct 30 21:24:53.565961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.565929    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"]
Oct 30 21:24:53.576503 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.576479    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b8868c9c7-xfdrb"]
Oct 30 21:24:53.989114 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:53.989076    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2916c977-34fa-4771-843d-1b999247b638" path="/var/lib/kubelet/pods/2916c977-34fa-4771-843d-1b999247b638/volumes"
Oct 30 21:24:54.201489 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.201452    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"]
Oct 30 21:24:54.202498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.202464    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing"
Oct 30 21:24:54.202498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.202493    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing"
Oct 30 21:24:54.202678 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.202648    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="2916c977-34fa-4771-843d-1b999247b638" containerName="payload-processing"
Oct 30 21:24:54.208213 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.208186    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.234943 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.234917    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"]
Oct 30 21:24:54.255669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.255590    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:54.268982 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.268952    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:54.339380 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339346    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.339586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339489    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.339586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339544    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.339740 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339677    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.339810 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339788    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.339894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339878    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.339954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339903    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.339954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339932    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.340025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.339996    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bqmz\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-kube-api-access-2bqmz\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.440798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.440764    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.440993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.440829    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.440993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.440912    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.440993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.440968    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441009    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441034    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441077    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441112    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bqmz\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-kube-api-access-2bqmz\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441197    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441235    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441309    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441385    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441433    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.441780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.441760    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.443497 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.443475    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.443717 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.443700    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.463047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.463021    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bqmz\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-kube-api-access-2bqmz\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.463236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.463215    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.523283 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.523204    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:54.540332 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.540258    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:54.691916 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.691877    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"]
Oct 30 21:24:54.696808 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.696781    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.727838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.727803    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"]
Oct 30 21:24:54.897476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.897386    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82347ec2-49bc-45b7-83d7-4c57b829e416-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.897476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.897451    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.897651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.897584    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.897748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.897696    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.897908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.897882    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.897997 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.897961    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.898119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.898101    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.898249 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.898132    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.898326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.898266    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmrn\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-kube-api-access-zvmrn\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:54.901083 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:24:54.901053    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb843ce_d0fb_4da3_a06d_5b0fda65d732.slice/crio-d60217c06a4f84e351b0924615bd9d8999e7b296cd4126bbb3257e346919e9ae WatchSource:0}: Error finding container d60217c06a4f84e351b0924615bd9d8999e7b296cd4126bbb3257e346919e9ae: Status 404 returned error can't find the container with id d60217c06a4f84e351b0924615bd9d8999e7b296cd4126bbb3257e346919e9ae
Oct 30 21:24:54.919385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.907425    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"]
Oct 30 21:24:54.919693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.919644    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:54.919781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.919731    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:54.919781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:54.919774    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000021    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82347ec2-49bc-45b7-83d7-4c57b829e416-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000076    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000111    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000187    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000282    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000322    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000407    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000436    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000496    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmrn\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-kube-api-access-zvmrn\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.000840    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.001204    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.001993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.001575    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82347ec2-49bc-45b7-83d7-4c57b829e416-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.002692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.002128    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.002692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.002424    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.005129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.005083    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.006194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.005317    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.020706 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.020638    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.021568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.021541    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmrn\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-kube-api-access-zvmrn\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.317177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.316455    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:55.570239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.566445    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" event={"ID":"ddb843ce-d0fb-4da3-a06d-5b0fda65d732","Type":"ContainerStarted","Data":"08c8e0400f404a07ef0546a0af263cd4d2ed1aab347df4eae5ce23102b36952f"}
Oct 30 21:24:55.570239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.566510    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" event={"ID":"ddb843ce-d0fb-4da3-a06d-5b0fda65d732","Type":"ContainerStarted","Data":"d60217c06a4f84e351b0924615bd9d8999e7b296cd4126bbb3257e346919e9ae"}
Oct 30 21:24:55.694180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:55.668041    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podStartSLOduration=1.668019001 podStartE2EDuration="1.668019001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:55.631988999 +0000 UTC m=+895.761240457" watchObservedRunningTime="2026-10-30 21:24:55.668019001 +0000 UTC m=+895.797270460"
Oct 30 21:24:56.637674 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:56.635600    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:24:56.764333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:56.764289    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:24:56.764595 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:56.764563    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:24:56.929901 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:56.849331    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"]
Oct 30 21:24:56.929901 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:56.925764    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:56.954962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:56.954559    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.051178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.037113    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:57.051178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.037242    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:57.051178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.037295    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:24:57.092202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.091293    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.351046 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.349964    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.523842 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.523742    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:24:57.524539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.524090    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:24:57.885276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.844283    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.944992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.943715    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 21:24:57.944992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.944169    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:24:57.963175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.955249    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:57.971177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:57.963461    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:24:58.113396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.106752    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:24:58.403547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.386348    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:58.497584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.497544    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" event={"ID":"82347ec2-49bc-45b7-83d7-4c57b829e416","Type":"ContainerStarted","Data":"30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92"}
Oct 30 21:24:58.497892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.497873    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:24:58.498034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.498015    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" event={"ID":"82347ec2-49bc-45b7-83d7-4c57b829e416","Type":"ContainerStarted","Data":"4ccd17934fd3c809382f016158c8a340a7d80a54b21a88211b223891054e1f30"}
Oct 30 21:24:58.498200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.498184    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:24:58.548924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.548888    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:24:58.558205 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.553134    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:24:58.595178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:58.584335    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podStartSLOduration=4.584315405 podStartE2EDuration="4.584315405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:58.536784841 +0000 UTC m=+898.666036299" watchObservedRunningTime="2026-10-30 21:24:58.584315405 +0000 UTC m=+898.713566863"
Oct 30 21:24:59.252182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.196493    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:59.252182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.198370    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 21:24:59.252182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.199022    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:59.252182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.200008    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:24:59.360776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.360742    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:24:59.361009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.360992    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:24:59.435184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.410330    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:59.450180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.438973    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded"
Oct 30 21:24:59.501211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.446557    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:24:59.501211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.494735    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:24:59.533392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.533296    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:24:59.556179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:24:59.537239    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:25:00.230181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.229904    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:25:00.230388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.230281    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:25:00.255180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.231595    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:00.255180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.234225    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:00.374463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.374407    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:25:00.374754 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.374706    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:25:00.447475 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.444349    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:00.532665 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.531725    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:25:00.532665 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:00.531799    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:25:01.233188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.232504    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:01.233188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.233089    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:25:01.246806 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.242091    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:25:01.250419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.249338    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:01.321696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.321660    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:25:01.328408 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.328369    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:25:01.516345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.497342    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:25:01.559308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.524994    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:25:01.560629 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.560584    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:25:01.971182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:01.949842    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:25:02.033536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.033489    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:02.185179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.184801    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:25:02.236378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.236277    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:02.236870 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.236277    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:02.249799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.249766    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:25:02.329193 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.323313    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:25:02.329193 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.323385    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:25:02.346711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.345397    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:25:02.347924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.347888    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:25:02.533038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.532942    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:25:02.533038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:02.533012    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:25:03.318741 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:03.318693    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:25:03.319043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:03.318987    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:25:03.526678 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:03.526636    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:25:03.527129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:03.526712    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:25:04.318191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:04.317635    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:25:04.318191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:04.317696    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:25:04.523812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:04.523774    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:25:04.523967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:04.523828    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:25:05.318024 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:05.317989    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:25:05.318458 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:05.318053    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:25:05.523838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:05.523799    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:25:05.524015 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:05.523857    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:25:06.321627 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:06.321587    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:25:06.322145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:06.321652    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:25:06.528638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:06.528606    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:25:06.529106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:06.529074    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:25:06.529758 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:06.529738    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:25:07.361738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:07.361609    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:25:07.362911 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:07.362879    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:25:07.364286 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:07.364248    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:25:09.862094 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.861372    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"]
Oct 30 21:25:09.868187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.868133    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:09.886197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.886127    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:25:09.897286 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.897253    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9d93cf183177\""
Oct 30 21:25:09.897454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.897367    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:25:09.897454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.897426    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:25:09.899337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.899303    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-99d681ebef1b\""
Oct 30 21:25:09.899337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.899303    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:25:09.899556 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.899396    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ef793be6a5e4\""
Oct 30 21:25:09.899556 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.899466    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dmwj5\""
Oct 30 21:25:09.902575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:09.902548    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"]
Oct 30 21:25:10.000851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.000814    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-routing-overlay-volume\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.001024 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.000950    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-plugins-config-volume\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.001024 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.000997    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-provider-credentials\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.001229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.001201    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7p4m\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-kube-api-access-v7p4m\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.101983 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.101945    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-plugins-config-volume\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.102224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.102008    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-provider-credentials\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.102224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.102181    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7p4m\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-kube-api-access-v7p4m\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.102373 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.102316    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-routing-overlay-volume\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.104310 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.103729    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-routing-overlay-volume\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.104310 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.103871    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-plugins-config-volume\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.115113 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.115031    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-provider-credentials\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.121084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.121056    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7p4m\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-kube-api-access-v7p4m\") pod \"payload-processing-external-model-7db7468d7f-s5h79\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") " pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.192446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.192403    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:10.497033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:10.496999    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"]
Oct 30 21:25:10.501541 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:25:10.501510    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd1339e_30c6_4ba2_af97_27036fac8359.slice/crio-2c6d2204bf777998bcfcede9e5239db23fc7bc4d8836eca47e8ca5d0a6b4e3a6 WatchSource:0}: Error finding container 2c6d2204bf777998bcfcede9e5239db23fc7bc4d8836eca47e8ca5d0a6b4e3a6: Status 404 returned error can't find the container with id 2c6d2204bf777998bcfcede9e5239db23fc7bc4d8836eca47e8ca5d0a6b4e3a6
Oct 30 21:25:11.031353 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:25:11.031316    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:25:11.287202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:11.287091    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79" event={"ID":"9cd1339e-30c6-4ba2-af97-27036fac8359","Type":"ContainerStarted","Data":"8916301200c8f2038973f1386e6e188b149a7916044e9c8c9f309ff600aec9af"}
Oct 30 21:25:11.287202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:11.287180    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:11.287386 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:11.287207    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79" event={"ID":"9cd1339e-30c6-4ba2-af97-27036fac8359","Type":"ContainerStarted","Data":"2c6d2204bf777998bcfcede9e5239db23fc7bc4d8836eca47e8ca5d0a6b4e3a6"}
Oct 30 21:25:11.326610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:11.326563    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79" podStartSLOduration=2.326549513 podStartE2EDuration="2.326549513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:11.323313616 +0000 UTC m=+911.452565072" watchObservedRunningTime="2026-10-30 21:25:11.326549513 +0000 UTC m=+911.455800981"
Oct 30 21:25:22.320862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:22.320830    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:25:25.218446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.218415    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"]
Oct 30 21:25:25.226838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.226781    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.270212 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.270179    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"]
Oct 30 21:25:25.308327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.307814    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2tr\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-kube-api-access-nb2tr\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.308327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.307881    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.308327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.307924    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.308327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.308018    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.308327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.308044    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.308327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.308113    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.308327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.308200    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.309221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.308935    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432e6150-2d0b-4e0a-9514-08e0fec591d7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.309221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.309023    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.410597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.410556    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.410799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.410662    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432e6150-2d0b-4e0a-9514-08e0fec591d7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.410799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.410710    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.410799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.410741    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2tr\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-kube-api-access-nb2tr\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.410799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.410779    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.410820    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.410913    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.410940    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.411010    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.411150    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.411191    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.411379    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.411639    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432e6150-2d0b-4e0a-9514-08e0fec591d7-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.411663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.411660    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.413749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.413721    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.413856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.413834    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.429358 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.429336    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.429473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.429389    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2tr\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-kube-api-access-nb2tr\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:25.570231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:25.569520    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:26.450451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:26.446890    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"]
Oct 30 21:25:26.548819 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:25:26.540494    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432e6150_2d0b_4e0a_9514_08e0fec591d7.slice/crio-018f32d8ed12f06d74fbe893a8f50a2f4a3c99262c49481545e0eb1f585859b7 WatchSource:0}: Error finding container 018f32d8ed12f06d74fbe893a8f50a2f4a3c99262c49481545e0eb1f585859b7: Status 404 returned error can't find the container with id 018f32d8ed12f06d74fbe893a8f50a2f4a3c99262c49481545e0eb1f585859b7
Oct 30 21:25:26.551297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:26.551260    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:26.552324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:26.551582    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:26.552619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:26.552581    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:27.638523 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:27.637522    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" event={"ID":"432e6150-2d0b-4e0a-9514-08e0fec591d7","Type":"ContainerStarted","Data":"f600f84467eacb671c4e93d525adb96b913947a1f0a4bb9f72bc65b907546117"}
Oct 30 21:25:27.638523 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:27.637591    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" event={"ID":"432e6150-2d0b-4e0a-9514-08e0fec591d7","Type":"ContainerStarted","Data":"018f32d8ed12f06d74fbe893a8f50a2f4a3c99262c49481545e0eb1f585859b7"}
Oct 30 21:25:27.710180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:27.706032    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podStartSLOduration=2.7060122 podStartE2EDuration="2.7060122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:27.691501507 +0000 UTC m=+927.820752965" watchObservedRunningTime="2026-10-30 21:25:27.7060122 +0000 UTC m=+927.835263657"
Oct 30 21:25:28.058997 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.056114    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"]
Oct 30 21:25:28.109864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.109785    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.130534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.130505    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"]
Oct 30 21:25:28.167211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.160216    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-49738b-openshift-default-dockercfg-h4mdb\""
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.271773    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.271836    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.271933    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj88m\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-kube-api-access-lj88m\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.272006    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a233a4d2-3e10-4170-ac3f-d11fc6302225-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.272036    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.272071    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.272130    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.272215    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.287559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.272242    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.314475 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.314405    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"]
Oct 30 21:25:28.314813 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.314789    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"]
Oct 30 21:25:28.314964 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.314727    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.420962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.420916    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a233a4d2-3e10-4170-ac3f-d11fc6302225-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.483175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.482496    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.483175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.482548    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.483175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.482642    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.483175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.482729    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.483175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.482755    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.483175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.482890    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.483175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.482946    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.483672 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.483251    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj88m\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-kube-api-access-lj88m\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.503183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.484178    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.503183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.484421    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.503183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.421957    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a233a4d2-3e10-4170-ac3f-d11fc6302225-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.503183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.484494    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.503183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.485802    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.506293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.506262    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.524179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.519745    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.572528 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.570968    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj88m\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-kube-api-access-lj88m\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.573187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.572731    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:28.573187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.572841    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:25:28.573187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.572909    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:25:28.573895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.573870    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.585210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.584956    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.585210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.585057    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.585210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.585121    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmn6l\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-kube-api-access-kmn6l\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.586058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.585150    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.586058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.585674    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2824625-1259-44b3-beac-44087a9d94ad-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.586058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.585739    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.586058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.585772    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.586058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.585824    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.586058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.585862    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2824625-1259-44b3-beac-44087a9d94ad-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.635600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.635562    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:28.715905 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.715874    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.716672 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.716556    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmn6l\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-kube-api-access-kmn6l\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.724233 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.717221    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.737312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.735286    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.745413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.735079    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2824625-1259-44b3-beac-44087a9d94ad-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.746083 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.746057    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.753590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.746295    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.774064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.774024    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.774831 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.774708    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2824625-1259-44b3-beac-44087a9d94ad-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.775337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.775228    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.775997 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.746911    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.776467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.746670    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.776964 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.776935    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2824625-1259-44b3-beac-44087a9d94ad-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.788316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.788268    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.792019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.791960    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.811047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.811008    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmn6l\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-kube-api-access-kmn6l\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.828226 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.823750    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2824625-1259-44b3-beac-44087a9d94ad-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:28.828961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:28.828933    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:29.099633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.098287    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:29.450768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.450661    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"]
Oct 30 21:25:29.453831 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:25:29.453794    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda233a4d2_3e10_4170_ac3f_d11fc6302225.slice/crio-4dd00863b1cf5c35f71a1f842f3b476d718acb5c32513c8a1062188e1c94da03 WatchSource:0}: Error finding container 4dd00863b1cf5c35f71a1f842f3b476d718acb5c32513c8a1062188e1c94da03: Status 404 returned error can't find the container with id 4dd00863b1cf5c35f71a1f842f3b476d718acb5c32513c8a1062188e1c94da03
Oct 30 21:25:29.456418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.456390    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:29.456493 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.456455    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:29.456493 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.456485    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:29.574089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.573483    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:25:29.574089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.573531    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:25:29.598848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.598819    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"]
Oct 30 21:25:29.607508 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.607471    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:29.607597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.607558    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:29.607645 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.607598    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:25:29.895540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.894421    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" event={"ID":"a233a4d2-3e10-4170-ac3f-d11fc6302225","Type":"ContainerStarted","Data":"27870abff2caf76f51d74c4320fae1d2c12e01b61338385b35f130cd31b8725b"}
Oct 30 21:25:29.895540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.894493    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" event={"ID":"a233a4d2-3e10-4170-ac3f-d11fc6302225","Type":"ContainerStarted","Data":"4dd00863b1cf5c35f71a1f842f3b476d718acb5c32513c8a1062188e1c94da03"}
Oct 30 21:25:29.907907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.907798    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" event={"ID":"c2824625-1259-44b3-beac-44087a9d94ad","Type":"ContainerStarted","Data":"88af22afaac3bd04f6fb135addad52784145dbd1e91096f0b979b15d92c863b9"}
Oct 30 21:25:29.907907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.907856    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" event={"ID":"c2824625-1259-44b3-beac-44087a9d94ad","Type":"ContainerStarted","Data":"de764fa52bfa97b7a676192c2a86d94d269fc909a13e070e33e517ee351ce52e"}
Oct 30 21:25:29.938376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:29.938320    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podStartSLOduration=2.938302135 podStartE2EDuration="2.938302135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:29.934756445 +0000 UTC m=+930.064007915" watchObservedRunningTime="2026-10-30 21:25:29.938302135 +0000 UTC m=+930.067553593"
Oct 30 21:25:30.004321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.001970    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podStartSLOduration=3.00194946 podStartE2EDuration="3.00194946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:29.983075697 +0000 UTC m=+930.112327147" watchObservedRunningTime="2026-10-30 21:25:30.00194946 +0000 UTC m=+930.131200916"
Oct 30 21:25:30.100365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.100328    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:30.103139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.103104    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:25:30.103444 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.103401    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:25:30.575448 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.575402    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:25:30.575753 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.575728    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:25:30.638397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.638250    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:30.638567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.638455    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:25:30.638567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:30.638505    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 21:25:31.165186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:31.161121    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:25:31.179182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:31.165720    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:25:31.634238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:31.608642    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:25:31.634238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:31.608707    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:25:31.660223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:31.636142    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:25:31.660223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:31.636219    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 21:25:32.122201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:32.122145    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:25:32.125316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:32.125281    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:25:32.264190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:32.261913    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:32.264190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:32.262048    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:32.595182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:32.574083    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:25:32.595182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:32.574170    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:25:32.659181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:32.658541    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:25:32.659181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:32.658600    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 21:25:33.110309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:33.110270    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:25:33.110623 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:33.110591    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:25:33.592189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:33.589610    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:25:33.592189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:33.589693    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:25:33.656180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:33.655825    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:25:33.656180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:33.655883    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 21:25:34.108191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:34.107549    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:25:34.108191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:34.107609    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:25:34.572551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:34.572512    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:25:34.572844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:34.572813    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 21:25:34.637141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:34.637109    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:25:34.637611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:34.637178    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 21:25:35.105180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:35.104668    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:25:35.105180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:35.104750    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:25:35.637442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:35.637410    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:25:35.637983 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:35.637957    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 21:25:36.102584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:36.102546    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:25:36.102890 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:36.102858    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:25:36.137834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:36.137803    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:36.138254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:36.138229    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:36.139184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:36.139149    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:25:36.636855 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:36.636810    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 21:25:36.637050 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:36.636877    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 21:25:37.099802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:37.099764    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 21:25:37.100243 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:37.099838    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 21:25:37.802851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:37.802797    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:37.803758 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:37.803730    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:37.804622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:37.804600    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:25:38.103048 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:38.102964    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:38.103507 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:38.103284    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:38.104114 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:38.104093    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:25:41.030136 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:25:41.030102    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:25:45.210180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:45.209257    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:45.240317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:45.240251    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:45.240992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:45.240574    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:45.242222 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:45.240619    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:45.276063 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:45.276010    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.217406 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.217362    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:25:46.230854 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.230708    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:25:46.426204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.259708    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.426204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.261000    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.426204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.261031    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.453230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.447755    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:46.469991 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.469904    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:25:46.497182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.471966    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:25:46.526295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:46.526259    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:25:47.005352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:47.005305    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:47.009298 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:47.009236    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:47.222297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:47.222246    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:25:47.421381 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:47.421318    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 21:25:47.421960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:47.421833    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:47.424276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:47.424239    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:47.451195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:47.451127    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:47.451892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:47.451865    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:25:48.243183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.234653    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.434313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.434261    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 21:25:48.456331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.456301    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:25:48.489182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.466873    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 21:25:48.489435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.477398    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:25:48.533364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.533265    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:48.533636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.533600    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:48.533927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.533905    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:48.534070 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:48.534051    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.011320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.011273    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:49.011603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.011577    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.293400 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.293279    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:25:49.435602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.435543    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.475197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.452298    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:25:49.498339 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.498291    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:49.498664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.498614    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.544250 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.519041    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.545075 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.544573    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:49.578524 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:49.547324    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 21:25:50.373137 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:50.373096    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:25:50.441638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:50.441598    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:50.465325 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:50.456757    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:25:50.552387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:50.552330    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:50.584195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:50.584130    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:50.881230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:50.881197    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:25:51.032179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:51.007205    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:25:51.039536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:51.039508    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:25:51.379289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:51.379242    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:51.564197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:51.560277    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:51.585277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:51.585232    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:51.644187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:51.644080    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:25:51.799768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:51.799739    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:25:52.134054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:25:52.134010    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:26:04.931859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:04.931811    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded"
Oct 30 21:26:10.344121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.344067    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"]
Oct 30 21:26:10.349673 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.349651    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.379336 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.379302    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"]
Oct 30 21:26:10.410106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410072    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.410324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410121    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.410324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410142    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.410324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410275    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.410324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410311    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dw27\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-kube-api-access-5dw27\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.410548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410471    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.410548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410517    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.410691 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410662    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.410842 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.410825    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512407    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512481    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512525    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512552    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512583    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512606    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dw27\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-kube-api-access-5dw27\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512719    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512760    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.512836    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.513250    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.513492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.513357    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.514501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.514415    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.515363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.515314    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.515767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.515737    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.516684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.516661    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.519418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.519393    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.533224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.533027    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.537309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.536549    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dw27\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-kube-api-access-5dw27\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:10.663400 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:10.663364    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:11.006944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:11.004341    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"]
Oct 30 21:26:11.040936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:11.040900    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:11.042141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:11.041685    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:11.042141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:11.041746    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:11.053606 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:26:11.053579    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:26:11.769039 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:11.767979    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" event={"ID":"7959c1d6-8d4c-478f-86e6-e4795aa0f110","Type":"ContainerStarted","Data":"c30f61be4b801a21c83e943cbd8b0a4752db3f6582aeac77720ccb74933c0d96"}
Oct 30 21:26:11.769039 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:11.768048    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" event={"ID":"7959c1d6-8d4c-478f-86e6-e4795aa0f110","Type":"ContainerStarted","Data":"bda6864b2f9687ba5be3c21d89a084b2753ec749ad4e99eda159b9af386b3115"}
Oct 30 21:26:11.825292 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:11.825242    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podStartSLOduration=1.825227941 podStartE2EDuration="1.825227941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:11.824271685 +0000 UTC m=+971.953523142" watchObservedRunningTime="2026-10-30 21:26:11.825227941 +0000 UTC m=+971.954479398"
Oct 30 21:26:12.681185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:12.680876    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:12.681185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:12.681001    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:26:12.681185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:12.681056    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 21:26:13.666595 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:13.666553    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:26:13.667275 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:13.667241    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 21:26:14.678180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:14.672488    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:26:14.678180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:14.672564    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 21:26:15.668663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:15.668627    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:26:15.668963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:15.668934    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 21:26:16.672344 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:16.672297    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:26:16.672975 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:16.672918    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 21:26:17.382181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:17.380287    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:17.444302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:17.444248    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:17.518182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:17.514277    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:17.551180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:17.550251    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:17.679182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:17.667599    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:26:17.679182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:17.667665    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 21:26:18.429181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.381252    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:18.429181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.382101    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:26:18.490201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.449335    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 21:26:18.490201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.467056    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:26:18.595182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.522221    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:18.596132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.570298    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:18.596662 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.596638    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:26:18.596841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.596826    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:26:18.734442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.723688    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:26:18.734442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:18.723757    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 21:26:19.058477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.033184    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.058477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.033673    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:19.058477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.033723    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.168179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.164968    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:19.168179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.165053    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.385243 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.382269    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:19.433268 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.433229    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:19.433517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.433488    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.456079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.456046    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:26:19.466182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.456962    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 21:26:19.466578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.457959    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:26:19.474882 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.466757    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:19.634726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.633621    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:26:19.635560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.635490    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:26:19.665011 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.664966    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 21:26:19.665254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:19.665027    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 21:26:21.092499 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:21.092466    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:21.093321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:21.093289    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:21.094948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:21.094926    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:26:24.976247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:24.976198    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"]
Oct 30 21:26:29.976994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:29.976950    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79" podUID="9cd1339e-30c6-4ba2-af97-27036fac8359" containerName="payload-processing" containerID="cri-o://8916301200c8f2038973f1386e6e188b149a7916044e9c8c9f309ff600aec9af" gracePeriod=25
Oct 30 21:26:30.211263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.211222    5886 generic.go:356] "Generic (PLEG): container finished" podID="9cd1339e-30c6-4ba2-af97-27036fac8359" containerID="8916301200c8f2038973f1386e6e188b149a7916044e9c8c9f309ff600aec9af" exitCode=0
Oct 30 21:26:30.211437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.211300    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79" event={"ID":"9cd1339e-30c6-4ba2-af97-27036fac8359","Type":"ContainerDied","Data":"8916301200c8f2038973f1386e6e188b149a7916044e9c8c9f309ff600aec9af"}
Oct 30 21:26:30.245844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.245785    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:26:30.358122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.358088    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-kube-api-access-v7p4m\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-kube-api-access-v7p4m\") pod \"9cd1339e-30c6-4ba2-af97-27036fac8359\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") "
Oct 30 21:26:30.358329 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.358191    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-plugins-config-volume\") pod \"9cd1339e-30c6-4ba2-af97-27036fac8359\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") "
Oct 30 21:26:30.358329 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.358220    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-routing-overlay-volume\") pod \"9cd1339e-30c6-4ba2-af97-27036fac8359\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") "
Oct 30 21:26:30.358462 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.358338    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-provider-credentials\") pod \"9cd1339e-30c6-4ba2-af97-27036fac8359\" (UID: \"9cd1339e-30c6-4ba2-af97-27036fac8359\") "
Oct 30 21:26:30.358608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.358578    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-routing-overlay-volume" pod "9cd1339e-30c6-4ba2-af97-27036fac8359" (UID: "9cd1339e-30c6-4ba2-af97-27036fac8359"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:30.358703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.358634    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-plugins-config-volume" pod "9cd1339e-30c6-4ba2-af97-27036fac8359" (UID: "9cd1339e-30c6-4ba2-af97-27036fac8359"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:30.358980 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.358962    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:26:30.359031 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.358981    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9cd1339e-30c6-4ba2-af97-27036fac8359-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:26:30.360392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.360368    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-kube-api-access-v7p4m" pod "9cd1339e-30c6-4ba2-af97-27036fac8359" (UID: "9cd1339e-30c6-4ba2-af97-27036fac8359"). InnerVolumeSpecName "kube-api-access-v7p4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:30.360525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.360507    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-provider-credentials" pod "9cd1339e-30c6-4ba2-af97-27036fac8359" (UID: "9cd1339e-30c6-4ba2-af97-27036fac8359"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:30.461127 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.461091    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7p4m\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-kube-api-access-v7p4m\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:26:30.461127 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:30.461125    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9cd1339e-30c6-4ba2-af97-27036fac8359-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:26:31.218720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:31.218682    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"
Oct 30 21:26:31.218720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:31.218705    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79" event={"ID":"9cd1339e-30c6-4ba2-af97-27036fac8359","Type":"ContainerDied","Data":"2c6d2204bf777998bcfcede9e5239db23fc7bc4d8836eca47e8ca5d0a6b4e3a6"}
Oct 30 21:26:31.219419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:31.218771    5886 scope.go:122] "RemoveContainer" containerID="8916301200c8f2038973f1386e6e188b149a7916044e9c8c9f309ff600aec9af"
Oct 30 21:26:31.269887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:31.269848    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"]
Oct 30 21:26:31.280537 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:31.280511    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7db7468d7f-s5h79"]
Oct 30 21:26:32.003788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:32.003751    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd1339e-30c6-4ba2-af97-27036fac8359" path="/var/lib/kubelet/pods/9cd1339e-30c6-4ba2-af97-27036fac8359/volumes"
Oct 30 21:26:33.464293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:33.464244    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:33.501181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:33.499381    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:33.567313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:33.567265    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.012179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.011289    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.250691 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.250319    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:34.250691 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.250405    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.482295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.482246    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:26:34.483684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.483657    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:26:34.500300 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.500254    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.524191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.501686    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:26:34.524191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.510246    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.561185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.534394    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:34.561185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.534458    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.578180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.576350    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.578975 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.578951    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:26:34.793183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.734198    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6qx95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:34.793183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.734258    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:34.965812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:34.965766    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.092235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:35.012774    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.092235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:35.014899    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:26:35.276185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:35.275835    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.487351 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:35.487303    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:26:35.517712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:35.517277    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.519289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:35.519095    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:35.521493 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:35.521468    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:26:35.604180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:35.593190    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 21:26:36.071181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:36.050822    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:26:36.584258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:36.488962    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:36.584258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:36.524229    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 21:26:36.584258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:36.518250    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:36.615055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:36.597273    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.051312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:37.051263    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.191886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:37.191853    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:26:37.417187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:37.406279    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:26:37.569556 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:37.569525    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:26:37.569824 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:37.569808    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:26:37.569956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:37.524442    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:37.631489 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:37.631452    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:26:41.030472 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:26:41.030385    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:26:43.209826 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.209794    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"]
Oct 30 21:26:43.210764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.210743    5886 cpu_manager.go:414] "containerMap: removing container" podUID="9cd1339e-30c6-4ba2-af97-27036fac8359" containerName="payload-processing"
Oct 30 21:26:43.210889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.210767    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd1339e-30c6-4ba2-af97-27036fac8359" containerName="payload-processing"
Oct 30 21:26:43.210945 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.210911    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd1339e-30c6-4ba2-af97-27036fac8359" containerName="payload-processing"
Oct 30 21:26:43.215944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.215922    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.243525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.243494    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"]
Oct 30 21:26:43.277374 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.277335    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.277374 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.277386    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.277638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.277413    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.277638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.277603    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.277724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.277688    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.277758 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.277734    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.277891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.277870    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.278033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.277987    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbca04c-a3b0-4159-976c-dfa3683e1595-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.278033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.278017    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8m4s\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-kube-api-access-p8m4s\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379081    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbca04c-a3b0-4159-976c-dfa3683e1595-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379140    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8m4s\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-kube-api-access-p8m4s\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379296    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379325    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379350    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379413    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379446    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379563    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379678    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379752    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379798    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379822    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.379944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379904    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.380016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.379996    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbca04c-a3b0-4159-976c-dfa3683e1595-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.381855 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.381832    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.382065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.382050    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.396608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.396575    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8m4s\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-kube-api-access-p8m4s\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.397403 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.397384    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.536935 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.536854    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:43.857554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.857411    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"]
Oct 30 21:26:43.861264 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:26:43.861237    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbca04c_a3b0_4159_976c_dfa3683e1595.slice/crio-bd8b53f02018836b2b09874450fd9f9aff3d3580caf672c9a41009f389504ef1 WatchSource:0}: Error finding container bd8b53f02018836b2b09874450fd9f9aff3d3580caf672c9a41009f389504ef1: Status 404 returned error can't find the container with id bd8b53f02018836b2b09874450fd9f9aff3d3580caf672c9a41009f389504ef1
Oct 30 21:26:43.863731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.863698    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:43.863820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.863758    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:43.863820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:43.863787    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:26:44.494115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:44.494079    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" event={"ID":"ffbca04c-a3b0-4159-976c-dfa3683e1595","Type":"ContainerStarted","Data":"bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110"}
Oct 30 21:26:44.494529 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:44.494134    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" event={"ID":"ffbca04c-a3b0-4159-976c-dfa3683e1595","Type":"ContainerStarted","Data":"bd8b53f02018836b2b09874450fd9f9aff3d3580caf672c9a41009f389504ef1"}
Oct 30 21:26:44.537860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:44.537827    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:44.539631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:44.539596    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:26:44.539751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:44.539656    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:26:44.545571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:44.545524    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podStartSLOduration=1.545508707 podStartE2EDuration="1.545508707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:44.543669343 +0000 UTC m=+1004.672920799" watchObservedRunningTime="2026-10-30 21:26:44.545508707 +0000 UTC m=+1004.674760164"
Oct 30 21:26:45.537134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:45.537098    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:26:45.537564 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:45.537177    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:26:46.537527 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:46.537494    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:26:46.537945 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:46.537570    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:26:47.538094 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:47.538054    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:26:47.538544 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:47.538109    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:26:48.537784 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:48.537752    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:26:48.537966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:48.537812    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:26:49.538070 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:49.538029    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:26:49.538611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:49.538080    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:26:50.537189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:50.537118    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:26:50.537385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:50.537196    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:26:51.541277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:51.541245    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:51.541734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:51.541598    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:51.542367 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:51.542344    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 21:26:55.218648 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.218603    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"]
Oct 30 21:26:55.224471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.224440    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.247275 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.247245    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"distinct-2-w2-9d677d-kserve-self-signed-certs\""
Oct 30 21:26:55.247481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.247245    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"default-dockercfg-dxblp\""
Oct 30 21:26:55.249352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.249323    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"]
Oct 30 21:26:55.255442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.255420    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"kube-root-ca.crt\""
Oct 30 21:26:55.255554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.255451    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9d677d\"/\"openshift-service-ca.crt\""
Oct 30 21:26:55.363401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.363363    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-model-cache\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.363610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.363424    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-home\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.363686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.363600    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57rx\" (UniqueName: \"kubernetes.io/projected/45416ce3-5386-4cbe-a463-506a0e16c627-kube-api-access-r57rx\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.363766 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.363742    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45416ce3-5386-4cbe-a463-506a0e16c627-tls-certs\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.363840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.363826    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-dshm\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.363946 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.363928    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-tmp-dir\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.465396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.465344    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-home\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.465616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.465478    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57rx\" (UniqueName: \"kubernetes.io/projected/45416ce3-5386-4cbe-a463-506a0e16c627-kube-api-access-r57rx\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.465616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.465585    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45416ce3-5386-4cbe-a463-506a0e16c627-tls-certs\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.465736 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.465707    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-dshm\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.465862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.465833    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-home\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.466139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.466112    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-tmp-dir\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.466320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.466149    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-model-cache\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.466546 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.466522    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-tmp-dir\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.466770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.466747    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-model-cache\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.468102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.468076    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-dshm\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.468752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.468703    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45416ce3-5386-4cbe-a463-506a0e16c627-tls-certs\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.484079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.484026    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57rx\" (UniqueName: \"kubernetes.io/projected/45416ce3-5386-4cbe-a463-506a0e16c627-kube-api-access-r57rx\") pod \"distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") " pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:55.542082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:55.541770    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:26:56.239747 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:56.239704    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"]
Oct 30 21:26:56.660072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:56.660028    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" event={"ID":"45416ce3-5386-4cbe-a463-506a0e16c627","Type":"ContainerStarted","Data":"9063fce699ac11dd9b1f6422a36dce2b03b6f637cacd3742792e8a32f90e089b"}
Oct 30 21:26:57.240229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:57.235334    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:57.508242 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:57.492261    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:57.534302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:57.534245    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:57.606180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:57.605262    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:58.056347 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.056297    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:26:58.249181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.248262    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:58.249181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.249040    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:26:58.492988 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.492934    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:58.524769 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.523653    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:26:58.541181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.540364    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:58.541181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.540808    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:26:58.650312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.647357    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 21:26:58.650312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:58.648199    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:26:59.140184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:59.118726    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:59.140184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:59.119617    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:26:59.269227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:59.249468    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:59.436558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:59.436451    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:59.682371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:59.682316    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 21:26:59.683889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:59.683842    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:26:59.700225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:26:59.688912    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:00.169243 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:00.149926    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:27:00.298873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:00.298747    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:00.683987 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:00.682841    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:00.688568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:00.688222    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:27:00.793122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:00.793075    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:01.159196 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:01.152954    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:27:01.300588 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:01.300485    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:01.717364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:01.717310    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:01.736408 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:01.736359    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 21:27:01.795002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:01.794955    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:01.844694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:01.844662    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:27:02.033198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:02.031234    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:02.354354 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:02.354248    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:02.508289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:02.508255    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:27:02.517289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:02.517263    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:27:02.517539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:02.517523    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:27:02.517712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:02.517697    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:27:03.465080 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:03.465030    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" event={"ID":"45416ce3-5386-4cbe-a463-506a0e16c627","Type":"ContainerStarted","Data":"b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8"}
Oct 30 21:27:03.519351 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:03.519291    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podStartSLOduration=8.519273223999999 podStartE2EDuration="8.519273224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:03.515408463 +0000 UTC m=+1023.644659921" watchObservedRunningTime="2026-10-30 21:27:03.519273224 +0000 UTC m=+1023.648524682"
Oct 30 21:27:03.542479 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:03.542446    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:27:05.543387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:05.543354    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:27:05.556051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:05.556025    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:27:06.243972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.243938    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"]
Oct 30 21:27:06.248689 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.248668    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.277380 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.277348    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3f2f4ebe-openshift-default-dockercfg-6ffgv\""
Oct 30 21:27:06.279373 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.279347    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"]
Oct 30 21:27:06.289794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.289763    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-envoy\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.290033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.290009    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-podinfo\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.290628 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.290307    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-certs\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.290757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.290738    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-socket\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.292008 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.291984    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kr95\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-kube-api-access-8kr95\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.292232 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.292212    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-credential-socket\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.292371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.292354    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-data\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.292477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.292463    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istiod-ca-cert\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.292590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.292577    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-token\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.393981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.393936    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-podinfo\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394006    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-certs\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394112    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-socket\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394139    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kr95\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-kube-api-access-8kr95\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394225    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-credential-socket\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394249    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-data\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394275    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istiod-ca-cert\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394319    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-token\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394408    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-envoy\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394521    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-certs\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394533    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-socket\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.394893 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.394839    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-data\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.395226 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.395201    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istiod-ca-cert\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.395358 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.395333    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-credential-socket\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.396567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.396547    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-envoy\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.396736 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.396716    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-podinfo\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.413234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.413180    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-token\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.414248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.414221    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kr95\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-kube-api-access-8kr95\") pod \"e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") " pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.547640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.547556    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:27:06.576026 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.575992    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:06.917735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.917699    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"]
Oct 30 21:27:06.919960 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:27:06.919928    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6add9d4_c0a6_4f56_9c06_bf7a5832f006.slice/crio-61fc59cefc76f4b1cfdc10cd5d1d387b4b52a36f38e478fbb7ee8dd732202a02 WatchSource:0}: Error finding container 61fc59cefc76f4b1cfdc10cd5d1d387b4b52a36f38e478fbb7ee8dd732202a02: Status 404 returned error can't find the container with id 61fc59cefc76f4b1cfdc10cd5d1d387b4b52a36f38e478fbb7ee8dd732202a02
Oct 30 21:27:06.922957 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.922920    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:06.923042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.923005    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:06.923078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:06.923046    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:07.544689 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:07.544652    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" event={"ID":"d6add9d4-c0a6-4f56-9c06-bf7a5832f006","Type":"ContainerStarted","Data":"1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480"}
Oct 30 21:27:07.544894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:07.544708    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" event={"ID":"d6add9d4-c0a6-4f56-9c06-bf7a5832f006","Type":"ContainerStarted","Data":"61fc59cefc76f4b1cfdc10cd5d1d387b4b52a36f38e478fbb7ee8dd732202a02"}
Oct 30 21:27:07.576838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:07.576810    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:07.583278 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:07.583256    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:07.635008 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:07.634957    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" podStartSLOduration=1.634944204 podStartE2EDuration="1.634944204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:07.595137963 +0000 UTC m=+1027.724389420" watchObservedRunningTime="2026-10-30 21:27:07.634944204 +0000 UTC m=+1027.764195660"
Oct 30 21:27:08.563010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:08.562978    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:08.564001 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:08.563981    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:10.910232 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:10.910199    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"]
Oct 30 21:27:10.915740 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:10.915706    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:10.935063 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:10.935022    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"]
Oct 30 21:27:10.939064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:10.939037    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-659efa\""
Oct 30 21:27:10.941106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:10.941082    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-659efa-dockercfg-z62kr\""
Oct 30 21:27:10.962311 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:10.962275    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmr4r\" (UniqueName: \"kubernetes.io/projected/cc002299-b819-47e0-982d-cdcebf8a359f-kube-api-access-gmr4r\") pod \"payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7\" (UID: \"cc002299-b819-47e0-982d-cdcebf8a359f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:10.962449 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:10.962321    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc002299-b819-47e0-982d-cdcebf8a359f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7\" (UID: \"cc002299-b819-47e0-982d-cdcebf8a359f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:11.057777 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:27:11.057738    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:27:11.073483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:11.063907    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmr4r\" (UniqueName: \"kubernetes.io/projected/cc002299-b819-47e0-982d-cdcebf8a359f-kube-api-access-gmr4r\") pod \"payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7\" (UID: \"cc002299-b819-47e0-982d-cdcebf8a359f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:11.073483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:11.063967    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc002299-b819-47e0-982d-cdcebf8a359f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7\" (UID: \"cc002299-b819-47e0-982d-cdcebf8a359f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:11.073483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:11.064940    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc002299-b819-47e0-982d-cdcebf8a359f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7\" (UID: \"cc002299-b819-47e0-982d-cdcebf8a359f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:11.127654 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:11.127234    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmr4r\" (UniqueName: \"kubernetes.io/projected/cc002299-b819-47e0-982d-cdcebf8a359f-kube-api-access-gmr4r\") pod \"payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7\" (UID: \"cc002299-b819-47e0-982d-cdcebf8a359f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:11.231777 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:11.231681    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:11.570564 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:11.570471    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"]
Oct 30 21:27:11.576767 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:27:11.576740    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc002299_b819_47e0_982d_cdcebf8a359f.slice/crio-5f896959902e51bc433ce905beb385748e6d98217450a37ad4bd3aa895bdfdfb WatchSource:0}: Error finding container 5f896959902e51bc433ce905beb385748e6d98217450a37ad4bd3aa895bdfdfb: Status 404 returned error can't find the container with id 5f896959902e51bc433ce905beb385748e6d98217450a37ad4bd3aa895bdfdfb
Oct 30 21:27:11.589888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:11.589852    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7" event={"ID":"cc002299-b819-47e0-982d-cdcebf8a359f","Type":"ContainerStarted","Data":"5f896959902e51bc433ce905beb385748e6d98217450a37ad4bd3aa895bdfdfb"}
Oct 30 21:27:12.601230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.601192    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7" event={"ID":"cc002299-b819-47e0-982d-cdcebf8a359f","Type":"ContainerStarted","Data":"9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee"}
Oct 30 21:27:12.601693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.601396    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:12.672596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.672540    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7" podStartSLOduration=2.672526611 podStartE2EDuration="2.672526611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:12.669666516 +0000 UTC m=+1032.798917974" watchObservedRunningTime="2026-10-30 21:27:12.672526611 +0000 UTC m=+1032.801778068"
Oct 30 21:27:12.746388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.746355    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"]
Oct 30 21:27:12.750838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.750820    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:12.766447 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.766422    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3f2f4ebe\""
Oct 30 21:27:12.766668 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.766652    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3f2f4ebe-dockercfg-p47mt\""
Oct 30 21:27:12.782919 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.782889    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"]
Oct 30 21:27:12.794150 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.794114    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2948fc-2854-4751-bdef-a6fb80c8bad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh\" (UID: \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:12.794284 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.794222    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x68g\" (UniqueName: \"kubernetes.io/projected/8c2948fc-2854-4751-bdef-a6fb80c8bad1-kube-api-access-9x68g\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh\" (UID: \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:12.939389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.923258    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2948fc-2854-4751-bdef-a6fb80c8bad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh\" (UID: \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:12.939389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.923319    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x68g\" (UniqueName: \"kubernetes.io/projected/8c2948fc-2854-4751-bdef-a6fb80c8bad1-kube-api-access-9x68g\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh\" (UID: \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:12.939389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.924608    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2948fc-2854-4751-bdef-a6fb80c8bad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh\" (UID: \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:12.960210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:12.957362    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x68g\" (UniqueName: \"kubernetes.io/projected/8c2948fc-2854-4751-bdef-a6fb80c8bad1-kube-api-access-9x68g\") pod \"payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh\" (UID: \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:13.065204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:13.065016    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:13.397776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:13.397747    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"]
Oct 30 21:27:13.402102 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:27:13.402060    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2948fc_2854_4751_bdef_a6fb80c8bad1.slice/crio-64e341fa974660624d1487bbae0d37e2289f77fc2bbeb7c0897ee72b8703e23d WatchSource:0}: Error finding container 64e341fa974660624d1487bbae0d37e2289f77fc2bbeb7c0897ee72b8703e23d: Status 404 returned error can't find the container with id 64e341fa974660624d1487bbae0d37e2289f77fc2bbeb7c0897ee72b8703e23d
Oct 30 21:27:13.611584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:13.611549    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh" event={"ID":"8c2948fc-2854-4751-bdef-a6fb80c8bad1","Type":"ContainerStarted","Data":"2e4f31432441a49118a78372b552aba3072792d1b078bc881258ba30201e7211"}
Oct 30 21:27:13.611966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:13.611612    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh" event={"ID":"8c2948fc-2854-4751-bdef-a6fb80c8bad1","Type":"ContainerStarted","Data":"64e341fa974660624d1487bbae0d37e2289f77fc2bbeb7c0897ee72b8703e23d"}
Oct 30 21:27:13.611966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:13.611891    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:13.654934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:13.654837    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh" podStartSLOduration=1.654825301 podStartE2EDuration="1.654825301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:13.654299606 +0000 UTC m=+1033.783551060" watchObservedRunningTime="2026-10-30 21:27:13.654825301 +0000 UTC m=+1033.784076757"
Oct 30 21:27:18.540180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:18.539378    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:18.797907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:18.797766    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:19.619701 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.619667    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"]
Oct 30 21:27:19.624173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.624134    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.642507 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.642472    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"]
Oct 30 21:27:19.645750 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.645724    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjmv\" (UniqueName: \"kubernetes.io/projected/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-kube-api-access-vgjmv\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.646007 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.645988    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-openshift-service-ca\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.646090 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.646074    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-metrics-certs\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.646213 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.646188    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-maas-api-tls\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.649258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.649237    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-659efa-serving-cert\""
Oct 30 21:27:19.747898 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.747863    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-openshift-service-ca\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.748105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.747928    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-metrics-certs\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.748105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.747962    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-maas-api-tls\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.748105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.747997    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjmv\" (UniqueName: \"kubernetes.io/projected/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-kube-api-access-vgjmv\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.748582 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.748549    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-openshift-service-ca\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.750453 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.750431    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-metrics-certs\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.750554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.750480    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-maas-api-tls\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.766069 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.766038    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjmv\" (UniqueName: \"kubernetes.io/projected/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-kube-api-access-vgjmv\") pod \"maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:19.937748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:19.937708    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:20.276424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:20.276328    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"]
Oct 30 21:27:20.279935 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:27:20.279905    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3c13c9_c9e7_4c35_95c8_ef8377bf36fb.slice/crio-e8684885e4499e9bfa05ae4014f3f4f303aabf6edb1d3f49737742370cf0d062 WatchSource:0}: Error finding container e8684885e4499e9bfa05ae4014f3f4f303aabf6edb1d3f49737742370cf0d062: Status 404 returned error can't find the container with id e8684885e4499e9bfa05ae4014f3f4f303aabf6edb1d3f49737742370cf0d062
Oct 30 21:27:20.281517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:20.281495    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:27:20.705760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:20.705715    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh" event={"ID":"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb","Type":"ContainerStarted","Data":"e8684885e4499e9bfa05ae4014f3f4f303aabf6edb1d3f49737742370cf0d062"}
Oct 30 21:27:21.720917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:21.720881    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh" event={"ID":"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb","Type":"ContainerStarted","Data":"55b5ed3201eda292bd08e6950819661ab6107f2af8cb65a5ce7c7f466f38df80"}
Oct 30 21:27:21.721902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:21.721880    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:21.762594 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:21.762531    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh" podStartSLOduration=2.355394179 podStartE2EDuration="2.762512418s" totalImagesPullingTime="407.118239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:21.76090747 +0000 UTC m=+1041.890158925" watchObservedRunningTime="2026-10-30 21:27:21.762512418 +0000 UTC m=+1041.891763877"
Oct 30 21:27:23.615282 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:23.615247    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:27:24.622221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:24.622192    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:27:28.740488 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:28.740456    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:27:34.356581 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.356551    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"]
Oct 30 21:27:34.361696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.361668    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.375914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.375883    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:27:34.376044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.375997    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:27:34.376044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.376001    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:27:34.386933 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.386882    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l5v2k\""
Oct 30 21:27:34.392781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.392756    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"]
Oct 30 21:27:34.455732 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.455702    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.455876 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.455743    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265qb\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-kube-api-access-265qb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.455876 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.455825    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.455954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.455884    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f6ded-e8b1-4004-8c65-fa9aecaf195b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.455987 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.455954    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.456032 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.456020    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.456072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.456039    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.557347 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.557312    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.557347 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.557352    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f6ded-e8b1-4004-8c65-fa9aecaf195b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.557600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.557383    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.557600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.557484    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.557600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.557513    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.557780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.557727    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.557780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.557771    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265qb\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-kube-api-access-265qb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.558314 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.558286    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.559850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.559821    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.559966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.559882    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.560009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.559997    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f6ded-e8b1-4004-8c65-fa9aecaf195b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.560234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.560215    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.575298 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.575276    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.575818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.575800    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-265qb\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-kube-api-access-265qb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6cd5q\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:34.676791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:34.676756    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:35.008742 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:35.008716    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"]
Oct 30 21:27:35.011392 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:27:35.011361    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415f6ded_e8b1_4004_8c65_fa9aecaf195b.slice/crio-a79b25754ae6fd75006829ddcf2c34eeb7a5db7e5f5efd3d74fa6738a0d7b325 WatchSource:0}: Error finding container a79b25754ae6fd75006829ddcf2c34eeb7a5db7e5f5efd3d74fa6738a0d7b325: Status 404 returned error can't find the container with id a79b25754ae6fd75006829ddcf2c34eeb7a5db7e5f5efd3d74fa6738a0d7b325
Oct 30 21:27:35.936166 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:35.936127    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q" event={"ID":"415f6ded-e8b1-4004-8c65-fa9aecaf195b","Type":"ContainerStarted","Data":"a79b25754ae6fd75006829ddcf2c34eeb7a5db7e5f5efd3d74fa6738a0d7b325"}
Oct 30 21:27:39.122263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:39.122226    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:39.122692 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:39.122288    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:40.056237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:40.056136    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q" event={"ID":"415f6ded-e8b1-4004-8c65-fa9aecaf195b","Type":"ContainerStarted","Data":"08a44ff0d959ff1d27b29c52abf289a0d3c28054ebe6b2caa6df00d3455c1bf7"}
Oct 30 21:27:40.056776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:40.056747    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:40.097178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:40.086174    5886 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6cd5q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:27:40.097178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:40.086235    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q" podUID="415f6ded-e8b1-4004-8c65-fa9aecaf195b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:27:40.122283 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:40.121037    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q" podStartSLOduration=2.013555836 podStartE2EDuration="6.121022598s" totalImagesPullingTime="4.107466762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:40.113053846 +0000 UTC m=+1060.242305303" watchObservedRunningTime="2026-10-30 21:27:40.121022598 +0000 UTC m=+1060.250274054"
Oct 30 21:27:41.094115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:41.094088    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 21:27:41.228288 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:27:41.228247    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:27:47.163050 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:47.163004    5886 generic.go:356] "Generic (PLEG): container finished" podID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerID="1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480" exitCode=0
Oct 30 21:27:47.164026 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:47.163986    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" event={"ID":"d6add9d4-c0a6-4f56-9c06-bf7a5832f006","Type":"ContainerDied","Data":"1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480"}
Oct 30 21:27:47.166395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:47.166368    5886 scope.go:122] "RemoveContainer" containerID="1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480"
Oct 30 21:27:47.226224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:47.216135    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:47.226224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:47.216319    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:47.226224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:47.216421    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:27:48.197526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:48.197492    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" event={"ID":"d6add9d4-c0a6-4f56-9c06-bf7a5832f006","Type":"ContainerStarted","Data":"c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2"}
Oct 30 21:27:48.578700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:48.578600    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:48.579053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:48.579028    5886 patch_prober.go:29] interesting pod/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:27:48.579226 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:48.579201    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:27:49.578567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:49.578521    5886 patch_prober.go:29] interesting pod/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:27:49.579095 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:49.578587    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:27:50.577595 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:50.577556    5886 patch_prober.go:29] interesting pod/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:27:50.577873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:50.577845    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:27:51.581179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:51.580608    5886 patch_prober.go:29] interesting pod/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:27:51.581179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:51.580689    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:27:52.576607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:52.576564    5886 patch_prober.go:29] interesting pod/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 21:27:52.576902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:52.576873    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 21:27:53.563655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:53.563621    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:53.581964 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:53.581937    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:27:54.260304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:27:54.260277    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:28:11.030378 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:28:11.030294    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:28:32.367383 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:32.367340    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"]
Oct 30 21:28:37.368652 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.368515    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7" podUID="cc002299-b819-47e0-982d-cdcebf8a359f" containerName="payload-pre-processing" containerID="cri-o://9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee" gracePeriod=25
Oct 30 21:28:37.682592 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.682561    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:28:37.730802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.730769    5886 generic.go:356] "Generic (PLEG): container finished" podID="cc002299-b819-47e0-982d-cdcebf8a359f" containerID="9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee" exitCode=0
Oct 30 21:28:37.730970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.730814    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"
Oct 30 21:28:37.730970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.730841    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7" event={"ID":"cc002299-b819-47e0-982d-cdcebf8a359f","Type":"ContainerDied","Data":"9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee"}
Oct 30 21:28:37.730970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.730902    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7" event={"ID":"cc002299-b819-47e0-982d-cdcebf8a359f","Type":"ContainerDied","Data":"5f896959902e51bc433ce905beb385748e6d98217450a37ad4bd3aa895bdfdfb"}
Oct 30 21:28:37.730970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.730934    5886 scope.go:122] "RemoveContainer" containerID="9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee"
Oct 30 21:28:37.742719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.742693    5886 scope.go:122] "RemoveContainer" containerID="9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee"
Oct 30 21:28:37.743099 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:28:37.743073    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee\": container with ID starting with 9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee not found: ID does not exist" containerID="9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee"
Oct 30 21:28:37.743228 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.743104    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee"} err="failed to get container status \"9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee\": rpc error: code = NotFound desc = could not find container \"9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee\": container with ID starting with 9d2ed60010b5060038d5cfa0c16ec69b88587ddefbcc73cec007d1ff7061a3ee not found: ID does not exist"
Oct 30 21:28:37.761524 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.761497    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc002299-b819-47e0-982d-cdcebf8a359f-kube-api-access-gmr4r\" (UniqueName: \"kubernetes.io/projected/cc002299-b819-47e0-982d-cdcebf8a359f-kube-api-access-gmr4r\") pod \"cc002299-b819-47e0-982d-cdcebf8a359f\" (UID: \"cc002299-b819-47e0-982d-cdcebf8a359f\") "
Oct 30 21:28:37.761642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.761554    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc002299-b819-47e0-982d-cdcebf8a359f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc002299-b819-47e0-982d-cdcebf8a359f-plugins-config-volume\") pod \"cc002299-b819-47e0-982d-cdcebf8a359f\" (UID: \"cc002299-b819-47e0-982d-cdcebf8a359f\") "
Oct 30 21:28:37.761990 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.761958    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc002299-b819-47e0-982d-cdcebf8a359f-plugins-config-volume" pod "cc002299-b819-47e0-982d-cdcebf8a359f" (UID: "cc002299-b819-47e0-982d-cdcebf8a359f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:28:37.762408 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.762366    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc002299-b819-47e0-982d-cdcebf8a359f-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:28:37.772981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.772951    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc002299-b819-47e0-982d-cdcebf8a359f-kube-api-access-gmr4r" pod "cc002299-b819-47e0-982d-cdcebf8a359f" (UID: "cc002299-b819-47e0-982d-cdcebf8a359f"). InnerVolumeSpecName "kube-api-access-gmr4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:37.864480 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:37.864444    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmr4r\" (UniqueName: \"kubernetes.io/projected/cc002299-b819-47e0-982d-cdcebf8a359f-kube-api-access-gmr4r\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:28:38.080044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:38.078180    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"]
Oct 30 21:28:38.104269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:38.104226    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-659efa-758bc7c447-hp6b7"]
Oct 30 21:28:39.992136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:39.992095    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc002299-b819-47e0-982d-cdcebf8a359f" path="/var/lib/kubelet/pods/cc002299-b819-47e0-982d-cdcebf8a359f/volumes"
Oct 30 21:28:41.030127 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:28:41.030077    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:28:55.783189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:55.783137    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"]
Oct 30 21:28:55.784087 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:55.784052    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh" podUID="af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" containerName="maas-api" containerID="cri-o://55b5ed3201eda292bd08e6950819661ab6107f2af8cb65a5ce7c7f466f38df80" gracePeriod=30
Oct 30 21:28:55.926251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:55.926214    5886 generic.go:356] "Generic (PLEG): container finished" podID="af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" containerID="55b5ed3201eda292bd08e6950819661ab6107f2af8cb65a5ce7c7f466f38df80" exitCode=0
Oct 30 21:28:55.926423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:55.926294    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh" event={"ID":"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb","Type":"ContainerDied","Data":"55b5ed3201eda292bd08e6950819661ab6107f2af8cb65a5ce7c7f466f38df80"}
Oct 30 21:28:56.113170 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.113121    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:28:56.225783 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.225749    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-metrics-certs\") pod \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") "
Oct 30 21:28:56.225973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.225837    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-openshift-service-ca\") pod \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") "
Oct 30 21:28:56.226052 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.226011    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-kube-api-access-vgjmv\" (UniqueName: \"kubernetes.io/projected/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-kube-api-access-vgjmv\") pod \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") "
Oct 30 21:28:56.226239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.226213    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-maas-api-tls\") pod \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\" (UID: \"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb\") "
Oct 30 21:28:56.227148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.227115    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-openshift-service-ca" pod "af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" (UID: "af3c13c9-c9e7-4c35-95c8-ef8377bf36fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:28:56.228249 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.228218    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-metrics-certs" pod "af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" (UID: "af3c13c9-c9e7-4c35-95c8-ef8377bf36fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:28:56.228730 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.228703    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-kube-api-access-vgjmv" pod "af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" (UID: "af3c13c9-c9e7-4c35-95c8-ef8377bf36fb"). InnerVolumeSpecName "kube-api-access-vgjmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:28:56.228880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.228778    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-maas-api-tls" pod "af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" (UID: "af3c13c9-c9e7-4c35-95c8-ef8377bf36fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:28:56.328609 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.328571    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgjmv\" (UniqueName: \"kubernetes.io/projected/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-kube-api-access-vgjmv\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:28:56.328609 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.328610    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:28:56.328849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.328625    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:28:56.328849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.328640    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:28:56.797173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.797128    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"]
Oct 30 21:28:56.797786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.797770    5886 cpu_manager.go:414] "containerMap: removing container" podUID="af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" containerName="maas-api"
Oct 30 21:28:56.797786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.797786    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" containerName="maas-api"
Oct 30 21:28:56.797891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.797816    5886 cpu_manager.go:414] "containerMap: removing container" podUID="cc002299-b819-47e0-982d-cdcebf8a359f" containerName="payload-pre-processing"
Oct 30 21:28:56.797891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.797822    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc002299-b819-47e0-982d-cdcebf8a359f" containerName="payload-pre-processing"
Oct 30 21:28:56.797994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.797912    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc002299-b819-47e0-982d-cdcebf8a359f" containerName="payload-pre-processing"
Oct 30 21:28:56.797994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.797920    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" containerName="maas-api"
Oct 30 21:28:56.801840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.801822    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:56.821913 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.821883    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"]
Oct 30 21:28:56.829474 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.828773    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c6dab8-serving-cert\""
Oct 30 21:28:56.934798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.934768    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"
Oct 30 21:28:56.934981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.934795    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh" event={"ID":"af3c13c9-c9e7-4c35-95c8-ef8377bf36fb","Type":"ContainerDied","Data":"e8684885e4499e9bfa05ae4014f3f4f303aabf6edb1d3f49737742370cf0d062"}
Oct 30 21:28:56.934981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.934872    5886 scope.go:122] "RemoveContainer" containerID="55b5ed3201eda292bd08e6950819661ab6107f2af8cb65a5ce7c7f466f38df80"
Oct 30 21:28:56.937782 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.937758    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tfjf\" (UniqueName: \"kubernetes.io/projected/fb2c0971-1286-445c-9bee-2cf7e0af366e-kube-api-access-2tfjf\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:56.937978 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.937960    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-maas-api-tls\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:56.938141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.938120    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2c0971-1286-445c-9bee-2cf7e0af366e-openshift-service-ca\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:56.938239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.938191    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-metrics-certs\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:56.983602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.983576    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"]
Oct 30 21:28:56.987148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:56.987123    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659efa-6f97c6cb49-n5gsh"]
Oct 30 21:28:57.039042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.039004    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-maas-api-tls\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.039240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.039108    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2c0971-1286-445c-9bee-2cf7e0af366e-openshift-service-ca\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.039240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.039127    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-metrics-certs\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.039240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.039207    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tfjf\" (UniqueName: \"kubernetes.io/projected/fb2c0971-1286-445c-9bee-2cf7e0af366e-kube-api-access-2tfjf\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.039851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.039825    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2c0971-1286-445c-9bee-2cf7e0af366e-openshift-service-ca\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.041795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.041769    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-metrics-certs\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.041983 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.041958    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-maas-api-tls\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.063890 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.063816    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tfjf\" (UniqueName: \"kubernetes.io/projected/fb2c0971-1286-445c-9bee-2cf7e0af366e-kube-api-access-2tfjf\") pod \"maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.119464 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.119433    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.434514 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.434486    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"]
Oct 30 21:28:57.437414 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:28:57.437384    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2c0971_1286_445c_9bee_2cf7e0af366e.slice/crio-80f291b23a54c34dfc90b5d69d4ba0cfb4a13d184c7fc98d2e3ad0edb79f6111 WatchSource:0}: Error finding container 80f291b23a54c34dfc90b5d69d4ba0cfb4a13d184c7fc98d2e3ad0edb79f6111: Status 404 returned error can't find the container with id 80f291b23a54c34dfc90b5d69d4ba0cfb4a13d184c7fc98d2e3ad0edb79f6111
Oct 30 21:28:57.945516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.945467    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c" event={"ID":"fb2c0971-1286-445c-9bee-2cf7e0af366e","Type":"ContainerStarted","Data":"7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6"}
Oct 30 21:28:57.946030 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.945535    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c" event={"ID":"fb2c0971-1286-445c-9bee-2cf7e0af366e","Type":"ContainerStarted","Data":"80f291b23a54c34dfc90b5d69d4ba0cfb4a13d184c7fc98d2e3ad0edb79f6111"}
Oct 30 21:28:57.946227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.946150    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:28:57.983606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.983554    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c" podStartSLOduration=1.572780913 podStartE2EDuration="1.983539608s" totalImagesPullingTime="410.758695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:57.983063282 +0000 UTC m=+1138.112314742" watchObservedRunningTime="2026-10-30 21:28:57.983539608 +0000 UTC m=+1138.112791065"
Oct 30 21:28:57.990768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:28:57.990730    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3c13c9-c9e7-4c35-95c8-ef8377bf36fb" path="/var/lib/kubelet/pods/af3c13c9-c9e7-4c35-95c8-ef8377bf36fb/volumes"
Oct 30 21:29:03.991245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:03.991212    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"]
Oct 30 21:29:04.967251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:04.967221    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:29:08.478465 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.478424    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"]
Oct 30 21:29:08.483701 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.483678    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:08.501986 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.501962    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"]
Oct 30 21:29:08.507458 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.507431    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ab5774\""
Oct 30 21:29:08.609184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.592461    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ab5774-dockercfg-77669\""
Oct 30 21:29:08.646323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.646277    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cz8\" (UniqueName: \"kubernetes.io/projected/289e5160-3503-401d-a9b6-cad4bebdfdb2-kube-api-access-t4cz8\") pod \"payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7\" (UID: \"289e5160-3503-401d-a9b6-cad4bebdfdb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:08.646887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.646852    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e5160-3503-401d-a9b6-cad4bebdfdb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7\" (UID: \"289e5160-3503-401d-a9b6-cad4bebdfdb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:08.749061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.748969    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e5160-3503-401d-a9b6-cad4bebdfdb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7\" (UID: \"289e5160-3503-401d-a9b6-cad4bebdfdb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:08.749272 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.749118    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cz8\" (UniqueName: \"kubernetes.io/projected/289e5160-3503-401d-a9b6-cad4bebdfdb2-kube-api-access-t4cz8\") pod \"payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7\" (UID: \"289e5160-3503-401d-a9b6-cad4bebdfdb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:08.749804 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.749785    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e5160-3503-401d-a9b6-cad4bebdfdb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7\" (UID: \"289e5160-3503-401d-a9b6-cad4bebdfdb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:08.768073 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.768042    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cz8\" (UniqueName: \"kubernetes.io/projected/289e5160-3503-401d-a9b6-cad4bebdfdb2-kube-api-access-t4cz8\") pod \"payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7\" (UID: \"289e5160-3503-401d-a9b6-cad4bebdfdb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:08.799148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.799104    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:08.992019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:08.991972    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh" podUID="8c2948fc-2854-4751-bdef-a6fb80c8bad1" containerName="payload-pre-processing" containerID="cri-o://2e4f31432441a49118a78372b552aba3072792d1b078bc881258ba30201e7211" gracePeriod=25
Oct 30 21:29:09.170586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:09.170556    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"]
Oct 30 21:29:09.172565 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:29:09.172530    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289e5160_3503_401d_a9b6_cad4bebdfdb2.slice/crio-779ea6faffe495e69648e80725a426d227dd49567dd8aea93ea330104cce85d8 WatchSource:0}: Error finding container 779ea6faffe495e69648e80725a426d227dd49567dd8aea93ea330104cce85d8: Status 404 returned error can't find the container with id 779ea6faffe495e69648e80725a426d227dd49567dd8aea93ea330104cce85d8
Oct 30 21:29:10.093786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:10.093742    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7" event={"ID":"289e5160-3503-401d-a9b6-cad4bebdfdb2","Type":"ContainerStarted","Data":"2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0"}
Oct 30 21:29:10.094244 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:10.093820    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:10.094244 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:10.093844    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7" event={"ID":"289e5160-3503-401d-a9b6-cad4bebdfdb2","Type":"ContainerStarted","Data":"779ea6faffe495e69648e80725a426d227dd49567dd8aea93ea330104cce85d8"}
Oct 30 21:29:10.130497 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:10.130448    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7" podStartSLOduration=2.130436474 podStartE2EDuration="2.130436474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:10.129682641 +0000 UTC m=+1150.258934097" watchObservedRunningTime="2026-10-30 21:29:10.130436474 +0000 UTC m=+1150.259687960"
Oct 30 21:29:11.031273 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:29:11.031236    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:29:14.620976 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:14.620935    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh" podUID="8c2948fc-2854-4751-bdef-a6fb80c8bad1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:29:21.106909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.106876    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:29:21.885319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.885284    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"]
Oct 30 21:29:21.890906 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.890880    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:21.910740 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.910715    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"]
Oct 30 21:29:21.915254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.912143    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ab5774-serving-cert\""
Oct 30 21:29:21.965675 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.965642    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01af642c-0673-4425-a9ad-6de5d36bc3ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:21.965848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.965736    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:21.965848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.965759    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5sg\" (UniqueName: \"kubernetes.io/projected/01af642c-0673-4425-a9ad-6de5d36bc3ae-kube-api-access-dm5sg\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:21.965848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:21.965813    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-metrics-certs\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.067797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.067760    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-metrics-certs\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.068075 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.068054    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01af642c-0673-4425-a9ad-6de5d36bc3ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.068202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.068174    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.068309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.068202    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5sg\" (UniqueName: \"kubernetes.io/projected/01af642c-0673-4425-a9ad-6de5d36bc3ae-kube-api-access-dm5sg\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.068632 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:29:22.068609    5886 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-serving-cert: secret "maas-api-e2e-api-a-ab5774-serving-cert" not found
Oct 30 21:29:22.068766 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:29:22.068686    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls podName:01af642c-0673-4425-a9ad-6de5d36bc3ae nodeName:}" failed. No retries permitted until 2026-10-30 21:29:22.568664395 +0000 UTC m=+1162.697915833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls") pod "maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h" (UID: "01af642c-0673-4425-a9ad-6de5d36bc3ae") : secret "maas-api-e2e-api-a-ab5774-serving-cert" not found
Oct 30 21:29:22.069314 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.069278    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01af642c-0673-4425-a9ad-6de5d36bc3ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.071149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.071113    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-metrics-certs\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.089555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.089512    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5sg\" (UniqueName: \"kubernetes.io/projected/01af642c-0673-4425-a9ad-6de5d36bc3ae-kube-api-access-dm5sg\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.575928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.575888    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.578430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.578402    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls\") pod \"maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:22.811974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:22.811940    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:23.121028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:23.121001    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"]
Oct 30 21:29:23.123535 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:29:23.123504    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01af642c_0673_4425_a9ad_6de5d36bc3ae.slice/crio-76ab06c64bbc3cd469211da131a81ed15ef951ae4949b2814fa49e71b3eef7d2 WatchSource:0}: Error finding container 76ab06c64bbc3cd469211da131a81ed15ef951ae4949b2814fa49e71b3eef7d2: Status 404 returned error can't find the container with id 76ab06c64bbc3cd469211da131a81ed15ef951ae4949b2814fa49e71b3eef7d2
Oct 30 21:29:23.201811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:23.201770    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h" event={"ID":"01af642c-0673-4425-a9ad-6de5d36bc3ae","Type":"ContainerStarted","Data":"76ab06c64bbc3cd469211da131a81ed15ef951ae4949b2814fa49e71b3eef7d2"}
Oct 30 21:29:24.210369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:24.210330    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h" event={"ID":"01af642c-0673-4425-a9ad-6de5d36bc3ae","Type":"ContainerStarted","Data":"f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71"}
Oct 30 21:29:24.210891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:24.210451    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:24.249575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:24.249519    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h" podStartSLOduration=2.90910498 podStartE2EDuration="3.249506415s" totalImagesPullingTime="340.401435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:24.247521916 +0000 UTC m=+1164.376773373" watchObservedRunningTime="2026-10-30 21:29:24.249506415 +0000 UTC m=+1164.378757894"
Oct 30 21:29:24.621415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:24.621312    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh" podUID="8c2948fc-2854-4751-bdef-a6fb80c8bad1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:29:29.248618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.248537    5886 generic.go:356] "Generic (PLEG): container finished" podID="8c2948fc-2854-4751-bdef-a6fb80c8bad1" containerID="2e4f31432441a49118a78372b552aba3072792d1b078bc881258ba30201e7211" exitCode=0
Oct 30 21:29:29.248618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.248598    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh" event={"ID":"8c2948fc-2854-4751-bdef-a6fb80c8bad1","Type":"ContainerDied","Data":"2e4f31432441a49118a78372b552aba3072792d1b078bc881258ba30201e7211"}
Oct 30 21:29:29.274682 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.274652    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:29:29.387483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.387447    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2948fc-2854-4751-bdef-a6fb80c8bad1-kube-api-access-9x68g\" (UniqueName: \"kubernetes.io/projected/8c2948fc-2854-4751-bdef-a6fb80c8bad1-kube-api-access-9x68g\") pod \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\" (UID: \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\") "
Oct 30 21:29:29.387726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.387706    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2948fc-2854-4751-bdef-a6fb80c8bad1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2948fc-2854-4751-bdef-a6fb80c8bad1-plugins-config-volume\") pod \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\" (UID: \"8c2948fc-2854-4751-bdef-a6fb80c8bad1\") "
Oct 30 21:29:29.388106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.388076    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2948fc-2854-4751-bdef-a6fb80c8bad1-plugins-config-volume" pod "8c2948fc-2854-4751-bdef-a6fb80c8bad1" (UID: "8c2948fc-2854-4751-bdef-a6fb80c8bad1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:29.388362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.388342    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c2948fc-2854-4751-bdef-a6fb80c8bad1-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:29:29.389697 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.389665    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2948fc-2854-4751-bdef-a6fb80c8bad1-kube-api-access-9x68g" pod "8c2948fc-2854-4751-bdef-a6fb80c8bad1" (UID: "8c2948fc-2854-4751-bdef-a6fb80c8bad1"). InnerVolumeSpecName "kube-api-access-9x68g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:29.490513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:29.490473    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x68g\" (UniqueName: \"kubernetes.io/projected/8c2948fc-2854-4751-bdef-a6fb80c8bad1-kube-api-access-9x68g\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:29:30.223312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:30.223278    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:29:30.257510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:30.257434    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"
Oct 30 21:29:30.257510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:30.257473    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh" event={"ID":"8c2948fc-2854-4751-bdef-a6fb80c8bad1","Type":"ContainerDied","Data":"64e341fa974660624d1487bbae0d37e2289f77fc2bbeb7c0897ee72b8703e23d"}
Oct 30 21:29:30.257943 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:30.257528    5886 scope.go:122] "RemoveContainer" containerID="2e4f31432441a49118a78372b552aba3072792d1b078bc881258ba30201e7211"
Oct 30 21:29:30.316201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:30.316130    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"]
Oct 30 21:29:30.320651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:30.320618    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3f2f4ebe-5d4fb9f59f-cc2nh"]
Oct 30 21:29:31.992029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:31.991991    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2948fc-2854-4751-bdef-a6fb80c8bad1" path="/var/lib/kubelet/pods/8c2948fc-2854-4751-bdef-a6fb80c8bad1/volumes"
Oct 30 21:29:34.256461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:34.255526    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:29:34.256461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:34.255602    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:34.385951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:34.385424    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-6qx95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:29:34.385951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:29:34.385482    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:29:41.029816 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:29:41.029729    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:30:11.029610 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:30:11.029564    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:30:40.227926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:40.227886    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"]
Oct 30 21:30:41.038248 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:30:41.038211    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:30:45.228317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:45.228276    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7" podUID="289e5160-3503-401d-a9b6-cad4bebdfdb2" containerName="payload-pre-processing" containerID="cri-o://2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0" gracePeriod=25
Oct 30 21:30:45.536569 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:45.536538    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:30:45.596847 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:45.596814    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289e5160-3503-401d-a9b6-cad4bebdfdb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e5160-3503-401d-a9b6-cad4bebdfdb2-plugins-config-volume\") pod \"289e5160-3503-401d-a9b6-cad4bebdfdb2\" (UID: \"289e5160-3503-401d-a9b6-cad4bebdfdb2\") "
Oct 30 21:30:45.597024 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:45.596873    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289e5160-3503-401d-a9b6-cad4bebdfdb2-kube-api-access-t4cz8\" (UniqueName: \"kubernetes.io/projected/289e5160-3503-401d-a9b6-cad4bebdfdb2-kube-api-access-t4cz8\") pod \"289e5160-3503-401d-a9b6-cad4bebdfdb2\" (UID: \"289e5160-3503-401d-a9b6-cad4bebdfdb2\") "
Oct 30 21:30:45.597254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:45.597228    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289e5160-3503-401d-a9b6-cad4bebdfdb2-plugins-config-volume" pod "289e5160-3503-401d-a9b6-cad4bebdfdb2" (UID: "289e5160-3503-401d-a9b6-cad4bebdfdb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:45.597563 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:45.597537    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e5160-3503-401d-a9b6-cad4bebdfdb2-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:30:45.599237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:45.599211    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289e5160-3503-401d-a9b6-cad4bebdfdb2-kube-api-access-t4cz8" pod "289e5160-3503-401d-a9b6-cad4bebdfdb2" (UID: "289e5160-3503-401d-a9b6-cad4bebdfdb2"). InnerVolumeSpecName "kube-api-access-t4cz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:45.699049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:45.699008    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4cz8\" (UniqueName: \"kubernetes.io/projected/289e5160-3503-401d-a9b6-cad4bebdfdb2-kube-api-access-t4cz8\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:30:46.057634 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.057598    5886 generic.go:356] "Generic (PLEG): container finished" podID="289e5160-3503-401d-a9b6-cad4bebdfdb2" containerID="2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0" exitCode=0
Oct 30 21:30:46.057795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.057659    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"
Oct 30 21:30:46.057795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.057690    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7" event={"ID":"289e5160-3503-401d-a9b6-cad4bebdfdb2","Type":"ContainerDied","Data":"2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0"}
Oct 30 21:30:46.057795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.057763    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7" event={"ID":"289e5160-3503-401d-a9b6-cad4bebdfdb2","Type":"ContainerDied","Data":"779ea6faffe495e69648e80725a426d227dd49567dd8aea93ea330104cce85d8"}
Oct 30 21:30:46.057795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.057794    5886 scope.go:122] "RemoveContainer" containerID="2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0"
Oct 30 21:30:46.068084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.068068    5886 scope.go:122] "RemoveContainer" containerID="2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0"
Oct 30 21:30:46.068422 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:30:46.068398    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0\": container with ID starting with 2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0 not found: ID does not exist" containerID="2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0"
Oct 30 21:30:46.068518 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.068426    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0"} err="failed to get container status \"2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0\": rpc error: code = NotFound desc = could not find container \"2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0\": container with ID starting with 2b693d12b55451b8611b099afe61ec4a82f5bf51e841a97f1ef6e736cdf5dbf0 not found: ID does not exist"
Oct 30 21:30:46.109166 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.109131    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"]
Oct 30 21:30:46.120236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:46.120201    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab5774-66f7d64f79-jmvw7"]
Oct 30 21:30:47.989203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:30:47.989170    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289e5160-3503-401d-a9b6-cad4bebdfdb2" path="/var/lib/kubelet/pods/289e5160-3503-401d-a9b6-cad4bebdfdb2/volumes"
Oct 30 21:31:01.186997 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.186963    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"]
Oct 30 21:31:01.187540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.187330    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c" podUID="fb2c0971-1286-445c-9bee-2cf7e0af366e" containerName="maas-api" containerID="cri-o://7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6" gracePeriod=30
Oct 30 21:31:01.539105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.539072    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:31:01.569813 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.569780    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"]
Oct 30 21:31:01.570011 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.569989    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h" podUID="01af642c-0673-4425-a9ad-6de5d36bc3ae" containerName="maas-api" containerID="cri-o://f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71" gracePeriod=30
Oct 30 21:31:01.746185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.745422    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-maas-api-tls\") pod \"fb2c0971-1286-445c-9bee-2cf7e0af366e\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") "
Oct 30 21:31:01.746185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.745605    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-metrics-certs\") pod \"fb2c0971-1286-445c-9bee-2cf7e0af366e\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") "
Oct 30 21:31:01.746185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.745731    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2c0971-1286-445c-9bee-2cf7e0af366e-kube-api-access-2tfjf\" (UniqueName: \"kubernetes.io/projected/fb2c0971-1286-445c-9bee-2cf7e0af366e-kube-api-access-2tfjf\") pod \"fb2c0971-1286-445c-9bee-2cf7e0af366e\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") "
Oct 30 21:31:01.746185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.745759    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2c0971-1286-445c-9bee-2cf7e0af366e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2c0971-1286-445c-9bee-2cf7e0af366e-openshift-service-ca\") pod \"fb2c0971-1286-445c-9bee-2cf7e0af366e\" (UID: \"fb2c0971-1286-445c-9bee-2cf7e0af366e\") "
Oct 30 21:31:01.766284 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.760193    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-metrics-certs" pod "fb2c0971-1286-445c-9bee-2cf7e0af366e" (UID: "fb2c0971-1286-445c-9bee-2cf7e0af366e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:01.766284 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.760363    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2c0971-1286-445c-9bee-2cf7e0af366e-openshift-service-ca" pod "fb2c0971-1286-445c-9bee-2cf7e0af366e" (UID: "fb2c0971-1286-445c-9bee-2cf7e0af366e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:01.805201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.786318    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-maas-api-tls" pod "fb2c0971-1286-445c-9bee-2cf7e0af366e" (UID: "fb2c0971-1286-445c-9bee-2cf7e0af366e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:01.805201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.786684    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2c0971-1286-445c-9bee-2cf7e0af366e-kube-api-access-2tfjf" pod "fb2c0971-1286-445c-9bee-2cf7e0af366e" (UID: "fb2c0971-1286-445c-9bee-2cf7e0af366e"). InnerVolumeSpecName "kube-api-access-2tfjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:01.853131 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.853097    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:01.853131 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.853129    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2c0971-1286-445c-9bee-2cf7e0af366e-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:01.853415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.853143    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tfjf\" (UniqueName: \"kubernetes.io/projected/fb2c0971-1286-445c-9bee-2cf7e0af366e-kube-api-access-2tfjf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:01.853415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:01.853194    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb2c0971-1286-445c-9bee-2cf7e0af366e-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:02.068815 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.068785    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:31:02.161449 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.161410    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls\") pod \"01af642c-0673-4425-a9ad-6de5d36bc3ae\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") "
Oct 30 21:31:02.161658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.161475    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-metrics-certs\") pod \"01af642c-0673-4425-a9ad-6de5d36bc3ae\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") "
Oct 30 21:31:02.161658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.161592    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01af642c-0673-4425-a9ad-6de5d36bc3ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01af642c-0673-4425-a9ad-6de5d36bc3ae-openshift-service-ca\") pod \"01af642c-0673-4425-a9ad-6de5d36bc3ae\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") "
Oct 30 21:31:02.161658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.161632    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01af642c-0673-4425-a9ad-6de5d36bc3ae-kube-api-access-dm5sg\" (UniqueName: \"kubernetes.io/projected/01af642c-0673-4425-a9ad-6de5d36bc3ae-kube-api-access-dm5sg\") pod \"01af642c-0673-4425-a9ad-6de5d36bc3ae\" (UID: \"01af642c-0673-4425-a9ad-6de5d36bc3ae\") "
Oct 30 21:31:02.162025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.161993    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01af642c-0673-4425-a9ad-6de5d36bc3ae-openshift-service-ca" pod "01af642c-0673-4425-a9ad-6de5d36bc3ae" (UID: "01af642c-0673-4425-a9ad-6de5d36bc3ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:02.162550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.162514    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01af642c-0673-4425-a9ad-6de5d36bc3ae-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:02.164450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.164418    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-metrics-certs" pod "01af642c-0673-4425-a9ad-6de5d36bc3ae" (UID: "01af642c-0673-4425-a9ad-6de5d36bc3ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:02.164571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.164463    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls" pod "01af642c-0673-4425-a9ad-6de5d36bc3ae" (UID: "01af642c-0673-4425-a9ad-6de5d36bc3ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:02.164571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.164492    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01af642c-0673-4425-a9ad-6de5d36bc3ae-kube-api-access-dm5sg" pod "01af642c-0673-4425-a9ad-6de5d36bc3ae" (UID: "01af642c-0673-4425-a9ad-6de5d36bc3ae"). InnerVolumeSpecName "kube-api-access-dm5sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:02.238305 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.238265    5886 generic.go:356] "Generic (PLEG): container finished" podID="01af642c-0673-4425-a9ad-6de5d36bc3ae" containerID="f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71" exitCode=0
Oct 30 21:31:02.238801 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.238322    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"
Oct 30 21:31:02.238801 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.238359    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h" event={"ID":"01af642c-0673-4425-a9ad-6de5d36bc3ae","Type":"ContainerDied","Data":"f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71"}
Oct 30 21:31:02.238801 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.238442    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h" event={"ID":"01af642c-0673-4425-a9ad-6de5d36bc3ae","Type":"ContainerDied","Data":"76ab06c64bbc3cd469211da131a81ed15ef951ae4949b2814fa49e71b3eef7d2"}
Oct 30 21:31:02.238801 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.238479    5886 scope.go:122] "RemoveContainer" containerID="f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71"
Oct 30 21:31:02.242875 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.242838    5886 generic.go:356] "Generic (PLEG): container finished" podID="fb2c0971-1286-445c-9bee-2cf7e0af366e" containerID="7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6" exitCode=0
Oct 30 21:31:02.243038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.242974    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c" event={"ID":"fb2c0971-1286-445c-9bee-2cf7e0af366e","Type":"ContainerDied","Data":"7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6"}
Oct 30 21:31:02.243038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.243007    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c" event={"ID":"fb2c0971-1286-445c-9bee-2cf7e0af366e","Type":"ContainerDied","Data":"80f291b23a54c34dfc90b5d69d4ba0cfb4a13d184c7fc98d2e3ad0edb79f6111"}
Oct 30 21:31:02.243038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.243012    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"
Oct 30 21:31:02.280727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.264425    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5sg\" (UniqueName: \"kubernetes.io/projected/01af642c-0673-4425-a9ad-6de5d36bc3ae-kube-api-access-dm5sg\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:02.280727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.264455    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:02.280727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.264474    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01af642c-0673-4425-a9ad-6de5d36bc3ae-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:02.360177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.359283    5886 scope.go:122] "RemoveContainer" containerID="f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71"
Oct 30 21:31:02.367188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.365074    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"]
Oct 30 21:31:02.375176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.373135    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ab5774-54598c7d8c-2mc7h"]
Oct 30 21:31:02.385209 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:31:02.382271    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71\": container with ID starting with f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71 not found: ID does not exist" containerID="f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71"
Oct 30 21:31:02.385209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.382317    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71"} err="failed to get container status \"f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71\": rpc error: code = NotFound desc = could not find container \"f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71\": container with ID starting with f588bf4d7402df9546fa01af92e416b951fa8af78b108ec6471dcd3acdfbbb71 not found: ID does not exist"
Oct 30 21:31:02.385209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.382343    5886 scope.go:122] "RemoveContainer" containerID="7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6"
Oct 30 21:31:02.395785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.395745    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"]
Oct 30 21:31:02.401304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.400099    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c6dab8-6f478c474d-2jr7c"]
Oct 30 21:31:02.402993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.402964    5886 scope.go:122] "RemoveContainer" containerID="7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6"
Oct 30 21:31:02.403610 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:31:02.403583    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6\": container with ID starting with 7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6 not found: ID does not exist" containerID="7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6"
Oct 30 21:31:02.403760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:02.403734    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6"} err="failed to get container status \"7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6\": rpc error: code = NotFound desc = could not find container \"7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6\": container with ID starting with 7996011274b77f4c16b42c71d44bb00893981b44ff198002d50c712fbf0eb2d6 not found: ID does not exist"
Oct 30 21:31:03.990704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:03.990669    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01af642c-0673-4425-a9ad-6de5d36bc3ae" path="/var/lib/kubelet/pods/01af642c-0673-4425-a9ad-6de5d36bc3ae/volumes"
Oct 30 21:31:03.991277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:03.991259    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2c0971-1286-445c-9bee-2cf7e0af366e" path="/var/lib/kubelet/pods/fb2c0971-1286-445c-9bee-2cf7e0af366e/volumes"
Oct 30 21:31:10.169357 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:10.169317    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"]
Oct 30 21:31:10.169749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:10.169555    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy" containerID="cri-o://08c8e0400f404a07ef0546a0af263cd4d2ed1aab347df4eae5ce23102b36952f" gracePeriod=30
Oct 30 21:31:10.182141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:10.182105    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"]
Oct 30 21:31:10.182386 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:10.182352    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy" containerID="cri-o://643a11bbf99d93066ea208cdebe2ae32cc555756b1aa2be231c61e21330617a9" gracePeriod=30
Oct 30 21:31:10.195198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:10.190233    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"]
Oct 30 21:31:10.195198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:10.190482    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy" containerID="cri-o://40d1c4a907c618d7a4d213ced0e44d2aa56104315c0a17897756e6de4a584a9f" gracePeriod=30
Oct 30 21:31:10.195198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:10.192587    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"]
Oct 30 21:31:10.195198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:10.192784    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy" containerID="cri-o://b3929457a920d813c5986087872801a4c96a3d4bafe9cd2af7d612508468c4e6" gracePeriod=30
Oct 30 21:31:11.031794 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:31:11.031696    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:31:15.404195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.403834    5886 generic.go:356] "Generic (PLEG): container finished" podID="fcc63398-277a-464d-928d-35c1d17164bb" containerID="643a11bbf99d93066ea208cdebe2ae32cc555756b1aa2be231c61e21330617a9" exitCode=0
Oct 30 21:31:15.404195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.403950    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" event={"ID":"fcc63398-277a-464d-928d-35c1d17164bb","Type":"ContainerDied","Data":"643a11bbf99d93066ea208cdebe2ae32cc555756b1aa2be231c61e21330617a9"}
Oct 30 21:31:15.412797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.406264    5886 generic.go:356] "Generic (PLEG): container finished" podID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerID="b3929457a920d813c5986087872801a4c96a3d4bafe9cd2af7d612508468c4e6" exitCode=0
Oct 30 21:31:15.412797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.406338    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" event={"ID":"da668566-a5aa-4f42-a0b6-02c6829bcbe9","Type":"ContainerDied","Data":"b3929457a920d813c5986087872801a4c96a3d4bafe9cd2af7d612508468c4e6"}
Oct 30 21:31:15.412797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.408645    5886 generic.go:356] "Generic (PLEG): container finished" podID="b822225a-462c-4fdf-af75-c411a958b44f" containerID="40d1c4a907c618d7a4d213ced0e44d2aa56104315c0a17897756e6de4a584a9f" exitCode=0
Oct 30 21:31:15.412797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.408725    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" event={"ID":"b822225a-462c-4fdf-af75-c411a958b44f","Type":"ContainerDied","Data":"40d1c4a907c618d7a4d213ced0e44d2aa56104315c0a17897756e6de4a584a9f"}
Oct 30 21:31:15.412797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.410674    5886 generic.go:356] "Generic (PLEG): container finished" podID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerID="08c8e0400f404a07ef0546a0af263cd4d2ed1aab347df4eae5ce23102b36952f" exitCode=0
Oct 30 21:31:15.412797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.410755    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" event={"ID":"ddb843ce-d0fb-4da3-a06d-5b0fda65d732","Type":"ContainerDied","Data":"08c8e0400f404a07ef0546a0af263cd4d2ed1aab347df4eae5ce23102b36952f"}
Oct 30 21:31:15.508694 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.508336    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534072    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-data\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534125    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-kube-api-access-2bqmz\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-kube-api-access-2bqmz\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534272    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-podinfo\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534348    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-socket\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534392    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-credential-socket\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534422    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-token\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-token\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534498    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-envoy\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534538    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istiod-ca-cert\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.534872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.534597    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-certs\") pod \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\" (UID: \"ddb843ce-d0fb-4da3-a06d-5b0fda65d732\") "
Oct 30 21:31:15.536713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.535884    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-certs" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.536713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.536136    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-data" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.539903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.539613    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-kube-api-access-2bqmz" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "kube-api-access-2bqmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.554395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.552975    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-credential-socket" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.554395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.553823    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istiod-ca-cert" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:15.554395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.553896    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-socket" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.558553 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.558517    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-podinfo" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:31:15.560897 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.560862    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-envoy" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.564578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.564554    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-token" pod "ddb843ce-d0fb-4da3-a06d-5b0fda65d732" (UID: "ddb843ce-d0fb-4da3-a06d-5b0fda65d732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.605704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.605661    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:31:15.631077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.631045    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:31:15.636259 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636233    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-data\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.636394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636272    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-kube-api-access-n7n5d\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-kube-api-access-n7n5d\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.636394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636290    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-istio-token\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-istio-token\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.636394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636314    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-socket\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.636394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636348    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-socket\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.636394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636373    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-credential-socket\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.636669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636422    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-kube-api-access-zdbhk\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-kube-api-access-zdbhk\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.636669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636441    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-data\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.636669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636483    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-envoy\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.636669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636538    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-token\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-token\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.636669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636562    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-envoy\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.636669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636617    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcc63398-277a-464d-928d-35c1d17164bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc63398-277a-464d-928d-35c1d17164bb-istio-podinfo\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.636669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636648    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-podinfo\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.637028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636697    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc63398-277a-464d-928d-35c1d17164bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc63398-277a-464d-928d-35c1d17164bb-istiod-ca-cert\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.637028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636719    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-certs\") pod \"fcc63398-277a-464d-928d-35c1d17164bb\" (UID: \"fcc63398-277a-464d-928d-35c1d17164bb\") "
Oct 30 21:31:15.637028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636756    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-credential-socket\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.637028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636791    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-certs\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.637028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.636807    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istiod-ca-cert\") pod \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\" (UID: \"da668566-a5aa-4f42-a0b6-02c6829bcbe9\") "
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637259    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637293    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bqmz\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-kube-api-access-2bqmz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637308    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637322    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637337    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637350    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637363    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637376    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637389    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb843ce-d0fb-4da3-a06d-5b0fda65d732-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.637496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637462    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-socket" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.638083 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.637697    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-data" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.639347 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.639237    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-credential-socket" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.639450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.639348    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-socket" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.639610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.639571    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-credential-socket" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.639946 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.639732    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc63398-277a-464d-928d-35c1d17164bb-istiod-ca-cert" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:15.640400 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.640102    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-data" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.641136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.640717    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-certs" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.641136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.640822    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-kube-api-access-n7n5d" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "kube-api-access-n7n5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.641136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.640872    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-envoy" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.641350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.641237    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-istio-token" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.641492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.641459    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-certs" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.641854 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.641833    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istiod-ca-cert" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:15.642855 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.642822    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-envoy" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.643093 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.643058    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-kube-api-access-zdbhk" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "kube-api-access-zdbhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.643225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.643175    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcc63398-277a-464d-928d-35c1d17164bb-istio-podinfo" pod "fcc63398-277a-464d-928d-35c1d17164bb" (UID: "fcc63398-277a-464d-928d-35c1d17164bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:31:15.644712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.644684    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-podinfo" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:31:15.645123 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.645102    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-token" pod "da668566-a5aa-4f42-a0b6-02c6829bcbe9" (UID: "da668566-a5aa-4f42-a0b6-02c6829bcbe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.655328 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.655282    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:31:15.738015 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.737921    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-socket\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.738214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738062    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-istio-token\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-istio-token\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.738214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738095    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-kube-api-access-gmqtw\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-kube-api-access-gmqtw\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.738356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738226    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-socket" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.738356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738251    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-data\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.738356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738317    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-envoy\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.738504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738380    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b822225a-462c-4fdf-af75-c411a958b44f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b822225a-462c-4fdf-af75-c411a958b44f-istiod-ca-cert\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.738504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738444    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-credential-socket\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.738504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738487    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-certs\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.738667 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738518    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b822225a-462c-4fdf-af75-c411a958b44f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b822225a-462c-4fdf-af75-c411a958b44f-istio-podinfo\") pod \"b822225a-462c-4fdf-af75-c411a958b44f\" (UID: \"b822225a-462c-4fdf-af75-c411a958b44f\") "
Oct 30 21:31:15.739067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738516    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-data" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.739067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738713    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-credential-socket" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.739067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738816    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b822225a-462c-4fdf-af75-c411a958b44f-istiod-ca-cert" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:15.739067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.738923    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-certs" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739192    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739214    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739227    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739243    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7n5d\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-kube-api-access-n7n5d\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739259    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcc63398-277a-464d-928d-35c1d17164bb-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739272    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739285    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739298    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739311    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739326    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdbhk\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-kube-api-access-zdbhk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739341    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739355    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b822225a-462c-4fdf-af75-c411a958b44f-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.739368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739368    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739381    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739393    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739405    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739419    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739430    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739443    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcc63398-277a-464d-928d-35c1d17164bb-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739457    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da668566-a5aa-4f42-a0b6-02c6829bcbe9-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739469    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcc63398-277a-464d-928d-35c1d17164bb-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739482    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcc63398-277a-464d-928d-35c1d17164bb-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.740009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.739495    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da668566-a5aa-4f42-a0b6-02c6829bcbe9-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.741473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.741306    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b822225a-462c-4fdf-af75-c411a958b44f-istio-podinfo" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:31:15.741473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.741380    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-kube-api-access-gmqtw" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "kube-api-access-gmqtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.741473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.741457    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-envoy" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:15.741714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.741544    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-istio-token" pod "b822225a-462c-4fdf-af75-c411a958b44f" (UID: "b822225a-462c-4fdf-af75-c411a958b44f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:15.840988 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.840955    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.840988 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.840982    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmqtw\" (UniqueName: \"kubernetes.io/projected/b822225a-462c-4fdf-af75-c411a958b44f-kube-api-access-gmqtw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.840988 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.840994    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b822225a-462c-4fdf-af75-c411a958b44f-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:15.841227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:15.841003    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b822225a-462c-4fdf-af75-c411a958b44f-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:16.419084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.419056    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"
Oct 30 21:31:16.419084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.419067    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd" event={"ID":"fcc63398-277a-464d-928d-35c1d17164bb","Type":"ContainerDied","Data":"7212899f41caa0fdfb39454793becb303ea791f6758d7f0320aadcf71cae5e79"}
Oct 30 21:31:16.419632 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.419200    5886 scope.go:122] "RemoveContainer" containerID="643a11bbf99d93066ea208cdebe2ae32cc555756b1aa2be231c61e21330617a9"
Oct 30 21:31:16.420787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.420718    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv" event={"ID":"da668566-a5aa-4f42-a0b6-02c6829bcbe9","Type":"ContainerDied","Data":"0ddb78bf9c2044cf02e32ca95ecc9227e731beada861d5be62fcd1fa091b45c4"}
Oct 30 21:31:16.420900 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.420824    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"
Oct 30 21:31:16.422807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.422786    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"
Oct 30 21:31:16.422807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.422795    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf" event={"ID":"b822225a-462c-4fdf-af75-c411a958b44f","Type":"ContainerDied","Data":"8709797fdfdea5f2ad892c367e0608519d34b3c55d37a55dfc2bce14dbe6b258"}
Oct 30 21:31:16.424618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.424600    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"
Oct 30 21:31:16.424720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.424630    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w" event={"ID":"ddb843ce-d0fb-4da3-a06d-5b0fda65d732","Type":"ContainerDied","Data":"d60217c06a4f84e351b0924615bd9d8999e7b296cd4126bbb3257e346919e9ae"}
Oct 30 21:31:16.431648 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.431632    5886 scope.go:122] "RemoveContainer" containerID="b3929457a920d813c5986087872801a4c96a3d4bafe9cd2af7d612508468c4e6"
Oct 30 21:31:16.441521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.441500    5886 scope.go:122] "RemoveContainer" containerID="40d1c4a907c618d7a4d213ced0e44d2aa56104315c0a17897756e6de4a584a9f"
Oct 30 21:31:16.450399 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.450380    5886 scope.go:122] "RemoveContainer" containerID="08c8e0400f404a07ef0546a0af263cd4d2ed1aab347df4eae5ce23102b36952f"
Oct 30 21:31:16.475454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.475429    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"]
Oct 30 21:31:16.486648 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.486625    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-l7v4w"]
Oct 30 21:31:16.510416 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.510388    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"]
Oct 30 21:31:16.514057 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.514033    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-7x8tv"]
Oct 30 21:31:16.545223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.545194    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"]
Oct 30 21:31:16.553390 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.553365    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-g6kfd"]
Oct 30 21:31:16.588560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.588524    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"]
Oct 30 21:31:16.594464 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:16.594436    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5h6tf"]
Oct 30 21:31:17.246143 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246105    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"]
Oct 30 21:31:17.246663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246649    5886 cpu_manager.go:414] "containerMap: removing container" podUID="fb2c0971-1286-445c-9bee-2cf7e0af366e" containerName="maas-api"
Oct 30 21:31:17.246721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246664    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2c0971-1286-445c-9bee-2cf7e0af366e" containerName="maas-api"
Oct 30 21:31:17.246721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246675    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy"
Oct 30 21:31:17.246721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246680    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy"
Oct 30 21:31:17.246721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246700    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy"
Oct 30 21:31:17.246721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246706    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy"
Oct 30 21:31:17.246721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246714    5886 cpu_manager.go:414] "containerMap: removing container" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy"
Oct 30 21:31:17.246721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246718    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246733    5886 cpu_manager.go:414] "containerMap: removing container" podUID="289e5160-3503-401d-a9b6-cad4bebdfdb2" containerName="payload-pre-processing"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246738    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289e5160-3503-401d-a9b6-cad4bebdfdb2" containerName="payload-pre-processing"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246751    5886 cpu_manager.go:414] "containerMap: removing container" podUID="01af642c-0673-4425-a9ad-6de5d36bc3ae" containerName="maas-api"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246755    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01af642c-0673-4425-a9ad-6de5d36bc3ae" containerName="maas-api"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246780    5886 cpu_manager.go:414] "containerMap: removing container" podUID="8c2948fc-2854-4751-bdef-a6fb80c8bad1" containerName="payload-pre-processing"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246785    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2948fc-2854-4751-bdef-a6fb80c8bad1" containerName="payload-pre-processing"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246794    5886 cpu_manager.go:414] "containerMap: removing container" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246798    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246877    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2948fc-2854-4751-bdef-a6fb80c8bad1" containerName="payload-pre-processing"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246885    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="289e5160-3503-401d-a9b6-cad4bebdfdb2" containerName="payload-pre-processing"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246892    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2c0971-1286-445c-9bee-2cf7e0af366e" containerName="maas-api"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246898    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b822225a-462c-4fdf-af75-c411a958b44f" containerName="istio-proxy"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246905    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="01af642c-0673-4425-a9ad-6de5d36bc3ae" containerName="maas-api"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246913    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" containerName="istio-proxy"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246918    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" containerName="istio-proxy"
Oct 30 21:31:17.246938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.246926    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc63398-277a-464d-928d-35c1d17164bb" containerName="istio-proxy"
Oct 30 21:31:17.252133 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.252110    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:17.268556 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.268532    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c027ae-dockercfg-rmb8m\""
Oct 30 21:31:17.268556 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.268548    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c027ae\""
Oct 30 21:31:17.275066 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.273720    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"]
Oct 30 21:31:17.369178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.362046    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwnjz\" (UniqueName: \"kubernetes.io/projected/d48c76be-d415-4a41-aeaf-6e7522cffe64-kube-api-access-vwnjz\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt\" (UID: \"d48c76be-d415-4a41-aeaf-6e7522cffe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:17.369178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.362239    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48c76be-d415-4a41-aeaf-6e7522cffe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt\" (UID: \"d48c76be-d415-4a41-aeaf-6e7522cffe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:17.436201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.434682    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"]
Oct 30 21:31:17.465505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.465467    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48c76be-d415-4a41-aeaf-6e7522cffe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt\" (UID: \"d48c76be-d415-4a41-aeaf-6e7522cffe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:17.465849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.465827    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwnjz\" (UniqueName: \"kubernetes.io/projected/d48c76be-d415-4a41-aeaf-6e7522cffe64-kube-api-access-vwnjz\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt\" (UID: \"d48c76be-d415-4a41-aeaf-6e7522cffe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:17.466431 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.466399    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48c76be-d415-4a41-aeaf-6e7522cffe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt\" (UID: \"d48c76be-d415-4a41-aeaf-6e7522cffe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:17.467559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.467531    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"]
Oct 30 21:31:17.467675 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.467660    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:17.493591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.493561    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwnjz\" (UniqueName: \"kubernetes.io/projected/d48c76be-d415-4a41-aeaf-6e7522cffe64-kube-api-access-vwnjz\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt\" (UID: \"d48c76be-d415-4a41-aeaf-6e7522cffe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:17.566519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.566422    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:17.566921 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.566894    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x\" (UID: \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:17.566998 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.566961    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvmx2\" (UniqueName: \"kubernetes.io/projected/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-kube-api-access-kvmx2\") pod \"payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x\" (UID: \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:17.668338 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.668295    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x\" (UID: \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:17.668513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.668389    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvmx2\" (UniqueName: \"kubernetes.io/projected/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-kube-api-access-kvmx2\") pod \"payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x\" (UID: \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:17.669309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.669283    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x\" (UID: \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:17.701506 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.699595    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvmx2\" (UniqueName: \"kubernetes.io/projected/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-kube-api-access-kvmx2\") pod \"payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x\" (UID: \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:17.788298 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.788257    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:17.881531 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.880576    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"]
Oct 30 21:31:17.890829 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:31:17.890795    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48c76be_d415_4a41_aeaf_6e7522cffe64.slice/crio-27e91a0ece67453c610fd712bcb1629487802c8144b103f33bc1bccd2e63f9e4 WatchSource:0}: Error finding container 27e91a0ece67453c610fd712bcb1629487802c8144b103f33bc1bccd2e63f9e4: Status 404 returned error can't find the container with id 27e91a0ece67453c610fd712bcb1629487802c8144b103f33bc1bccd2e63f9e4
Oct 30 21:31:17.990645 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.990610    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b822225a-462c-4fdf-af75-c411a958b44f" path="/var/lib/kubelet/pods/b822225a-462c-4fdf-af75-c411a958b44f/volumes"
Oct 30 21:31:17.991307 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.991288    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da668566-a5aa-4f42-a0b6-02c6829bcbe9" path="/var/lib/kubelet/pods/da668566-a5aa-4f42-a0b6-02c6829bcbe9/volumes"
Oct 30 21:31:17.991890 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.991873    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb843ce-d0fb-4da3-a06d-5b0fda65d732" path="/var/lib/kubelet/pods/ddb843ce-d0fb-4da3-a06d-5b0fda65d732/volumes"
Oct 30 21:31:17.992985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:17.992960    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc63398-277a-464d-928d-35c1d17164bb" path="/var/lib/kubelet/pods/fcc63398-277a-464d-928d-35c1d17164bb/volumes"
Oct 30 21:31:18.095520 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.095488    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"]
Oct 30 21:31:18.097543 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:31:18.097514    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5dc2f5_15ce_4cf5_9a5e_0fab34e3518f.slice/crio-d95e0de856e7c19c43e59e4e1a472eca92e82f56e44dec584789a9b8a372ff04 WatchSource:0}: Error finding container d95e0de856e7c19c43e59e4e1a472eca92e82f56e44dec584789a9b8a372ff04: Status 404 returned error can't find the container with id d95e0de856e7c19c43e59e4e1a472eca92e82f56e44dec584789a9b8a372ff04
Oct 30 21:31:18.496000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.495963    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt" event={"ID":"d48c76be-d415-4a41-aeaf-6e7522cffe64","Type":"ContainerStarted","Data":"cd991e98e13a74221f12ef6ea0970bb1e1d228bb720403dbc81ddef8b55ea25d"}
Oct 30 21:31:18.496492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.496024    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt" event={"ID":"d48c76be-d415-4a41-aeaf-6e7522cffe64","Type":"ContainerStarted","Data":"27e91a0ece67453c610fd712bcb1629487802c8144b103f33bc1bccd2e63f9e4"}
Oct 30 21:31:18.496492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.496051    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:18.501200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.498729    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x" event={"ID":"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f","Type":"ContainerStarted","Data":"95caf17b0d165343da3b1a2505ae26b317db849a920f8875159f6fbd98e374c7"}
Oct 30 21:31:18.501200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.498828    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x" event={"ID":"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f","Type":"ContainerStarted","Data":"d95e0de856e7c19c43e59e4e1a472eca92e82f56e44dec584789a9b8a372ff04"}
Oct 30 21:31:18.501200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.499809    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:18.540071 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.540020    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt" podStartSLOduration=1.540007804 podStartE2EDuration="1.540007804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:18.538357154 +0000 UTC m=+1278.667608611" watchObservedRunningTime="2026-10-30 21:31:18.540007804 +0000 UTC m=+1278.669259260"
Oct 30 21:31:18.577857 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:18.577807    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x" podStartSLOduration=1.577793229 podStartE2EDuration="1.577793229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:18.574886341 +0000 UTC m=+1278.704137799" watchObservedRunningTime="2026-10-30 21:31:18.577793229 +0000 UTC m=+1278.707044686"
Oct 30 21:31:24.772749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.772715    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"]
Oct 30 21:31:24.777065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.777046    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.802907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.802872    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-229795-serving-cert\""
Oct 30 21:31:24.807202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.807175    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"]
Oct 30 21:31:24.868748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.868713    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8542a0-30ea-4343-a9a3-9c9c3567709f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.868924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.868761    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7wc\" (UniqueName: \"kubernetes.io/projected/0b8542a0-30ea-4343-a9a3-9c9c3567709f-kube-api-access-zc7wc\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.869006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.868983    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-metrics-certs\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.869095 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.869075    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.970781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.970742    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-metrics-certs\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.970979 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.970813    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.971055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.970979    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8542a0-30ea-4343-a9a3-9c9c3567709f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.971055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.971017    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7wc\" (UniqueName: \"kubernetes.io/projected/0b8542a0-30ea-4343-a9a3-9c9c3567709f-kube-api-access-zc7wc\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.971681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.971655    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8542a0-30ea-4343-a9a3-9c9c3567709f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.973953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.973921    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-metrics-certs\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.973953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.973934    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:24.990138 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:24.990103    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7wc\" (UniqueName: \"kubernetes.io/projected/0b8542a0-30ea-4343-a9a3-9c9c3567709f-kube-api-access-zc7wc\") pod \"maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:25.095727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:25.095394    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:25.424061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:25.424032    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"]
Oct 30 21:31:25.427848 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:31:25.427819    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8542a0_30ea_4343_a9a3_9c9c3567709f.slice/crio-86e30e28de1005979cd684fa1966a7aeab847436d139d33424be955ec7792b8e WatchSource:0}: Error finding container 86e30e28de1005979cd684fa1966a7aeab847436d139d33424be955ec7792b8e: Status 404 returned error can't find the container with id 86e30e28de1005979cd684fa1966a7aeab847436d139d33424be955ec7792b8e
Oct 30 21:31:25.554987 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:25.554953    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz" event={"ID":"0b8542a0-30ea-4343-a9a3-9c9c3567709f","Type":"ContainerStarted","Data":"86e30e28de1005979cd684fa1966a7aeab847436d139d33424be955ec7792b8e"}
Oct 30 21:31:26.575112 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:26.575076    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz" event={"ID":"0b8542a0-30ea-4343-a9a3-9c9c3567709f","Type":"ContainerStarted","Data":"ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c"}
Oct 30 21:31:26.575567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:26.575323    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:26.613576 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:26.613528    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz" podStartSLOduration=2.040200391 podStartE2EDuration="2.613512353s" totalImagesPullingTime="573.311962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:26.612796561 +0000 UTC m=+1286.742047996" watchObservedRunningTime="2026-10-30 21:31:26.613512353 +0000 UTC m=+1286.742763810"
Oct 30 21:31:29.510236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:29.510209    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:30.519291 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:30.519259    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:32.590864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:32.590833    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:31:39.160284 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:39.160242    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"]
Oct 30 21:31:39.160784 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:39.160511    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy" containerID="cri-o://c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2" gracePeriod=30
Oct 30 21:31:39.262745 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:39.262703    5886 patch_prober.go:29] interesting pod/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:31:39.262919 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:39.262816    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:31:40.650336 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.650300    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"]
Oct 30 21:31:40.676708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.676648    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.705496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.705460    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"]
Oct 30 21:31:40.708718 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.708689    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-18a043-openshift-default-dockercfg-5d2j2\""
Oct 30 21:31:40.793834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.793792    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istiod-ca-cert\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.794017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.793885    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-data\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.794017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.793987    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-envoy\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.794137 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.794079    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-socket\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.794198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.794171    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-credential-socket\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.794443 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.794417    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-podinfo\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.794526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.794463    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-certs\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.794526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.794498    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-token\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.794708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.794619    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kp5g\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-kube-api-access-9kp5g\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.896502 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896466    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-token\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.896709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896517    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kp5g\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-kube-api-access-9kp5g\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.896709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896586    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istiod-ca-cert\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.896709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896637    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-data\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.896709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896678    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-envoy\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.896925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896779    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-socket\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.896925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896847    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-credential-socket\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.897043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896959    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-podinfo\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.897043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.896998    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-certs\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.897801 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.897149    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-socket\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.900266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.900239    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-credential-socket\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.900392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.900318    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-certs\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.900392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.900358    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-envoy\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.900519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.900397    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-data\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.900904 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.900876    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istiod-ca-cert\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.902246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.902223    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-podinfo\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.915910 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.915868    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-token\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:40.916066 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:40.915977    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kp5g\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-kube-api-access-9kp5g\") pod \"e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") " pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:41.037195 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:31:41.034684    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:31:41.040242 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:41.040210    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:41.389193 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:41.389144    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"]
Oct 30 21:31:41.390901 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:31:41.390871    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcc8030_8b61_4bf2_aff3_423a4cf469f7.slice/crio-350fa8d81e161110900103e5374375b7bf5cc4fb00e088e7d7358b1575a07931 WatchSource:0}: Error finding container 350fa8d81e161110900103e5374375b7bf5cc4fb00e088e7d7358b1575a07931: Status 404 returned error can't find the container with id 350fa8d81e161110900103e5374375b7bf5cc4fb00e088e7d7358b1575a07931
Oct 30 21:31:41.393873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:41.393839    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:41.393974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:41.393923    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:41.393974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:41.393967    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:31:41.739819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:41.739776    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" event={"ID":"9fcc8030-8b61-4bf2-aff3-423a4cf469f7","Type":"ContainerStarted","Data":"8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9"}
Oct 30 21:31:41.740255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:41.739843    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" event={"ID":"9fcc8030-8b61-4bf2-aff3-423a4cf469f7","Type":"ContainerStarted","Data":"350fa8d81e161110900103e5374375b7bf5cc4fb00e088e7d7358b1575a07931"}
Oct 30 21:31:41.807589 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:41.807533    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" podStartSLOduration=1.807515101 podStartE2EDuration="1.807515101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:41.805619098 +0000 UTC m=+1301.934870559" watchObservedRunningTime="2026-10-30 21:31:41.807515101 +0000 UTC m=+1301.936766559"
Oct 30 21:31:42.040721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:42.040635    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:42.046672 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:42.046646    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:42.747020 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:42.746988    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:42.748072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:42.748055    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:31:44.445083 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.444987    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:31:44.546174 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546131    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-data\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.546357 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546227    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istiod-ca-cert\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.546357 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546288    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-certs\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.546357 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546320    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-kube-api-access-8kr95\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-kube-api-access-8kr95\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.546538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546401    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-envoy\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.546538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546423    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-data" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:44.546538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546436    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-token\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-token\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.546538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546463    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-credential-socket\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.546538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546529    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-socket\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.546818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546640    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-certs" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:44.546818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546654    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istiod-ca-cert" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:44.546818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.546694    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-podinfo\") pod \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\" (UID: \"d6add9d4-c0a6-4f56-9c06-bf7a5832f006\") "
Oct 30 21:31:44.547077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.547053    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-credential-socket" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:44.547297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.547275    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.547400 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.547302    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.547400 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.547317    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.547400 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.547330    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.547559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.547533    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-socket" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:44.549388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.549359    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-kube-api-access-8kr95" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "kube-api-access-8kr95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:44.549631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.549561    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-token" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:44.549802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.549774    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-envoy" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:44.549917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.549850    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-podinfo" pod "d6add9d4-c0a6-4f56-9c06-bf7a5832f006" (UID: "d6add9d4-c0a6-4f56-9c06-bf7a5832f006"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:31:44.648651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.648617    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.648651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.648648    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.648651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.648657    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kr95\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-kube-api-access-8kr95\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.648887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.648668    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.648887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.648676    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6add9d4-c0a6-4f56-9c06-bf7a5832f006-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:44.764711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.764619    5886 generic.go:356] "Generic (PLEG): container finished" podID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerID="c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2" exitCode=0
Oct 30 21:31:44.764866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.764701    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" event={"ID":"d6add9d4-c0a6-4f56-9c06-bf7a5832f006","Type":"ContainerDied","Data":"c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2"}
Oct 30 21:31:44.764866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.764724    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"
Oct 30 21:31:44.764866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.764787    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w" event={"ID":"d6add9d4-c0a6-4f56-9c06-bf7a5832f006","Type":"ContainerDied","Data":"61fc59cefc76f4b1cfdc10cd5d1d387b4b52a36f38e478fbb7ee8dd732202a02"}
Oct 30 21:31:44.764866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.764813    5886 scope.go:122] "RemoveContainer" containerID="c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2"
Oct 30 21:31:44.778967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.778947    5886 scope.go:122] "RemoveContainer" containerID="1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480"
Oct 30 21:31:44.790472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.789726    5886 scope.go:122] "RemoveContainer" containerID="c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2"
Oct 30 21:31:44.793221 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:31:44.791955    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2\": container with ID starting with c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2 not found: ID does not exist" containerID="c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2"
Oct 30 21:31:44.793221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.791989    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2"} err="failed to get container status \"c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2\": rpc error: code = NotFound desc = could not find container \"c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2\": container with ID starting with c7bc9dc7ce0d61ea0a94d909bba5100ebf282e67947cd0df698070370e7adcf2 not found: ID does not exist"
Oct 30 21:31:44.793221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.792012    5886 scope.go:122] "RemoveContainer" containerID="1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480"
Oct 30 21:31:44.793221 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:31:44.793051    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480\": container with ID starting with 1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480 not found: ID does not exist" containerID="1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480"
Oct 30 21:31:44.793221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.793077    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480"} err="failed to get container status \"1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480\": rpc error: code = NotFound desc = could not find container \"1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480\": container with ID starting with 1947183be9f3ca82d2d8c70adeca7240c61feea6443adefb25bbcf11ff846480 not found: ID does not exist"
Oct 30 21:31:44.816658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.816631    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"]
Oct 30 21:31:44.826501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:44.826478    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3f2f4ebe-openshift-default-5df95f56d9-f477w"]
Oct 30 21:31:45.991072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:45.991037    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" path="/var/lib/kubelet/pods/d6add9d4-c0a6-4f56-9c06-bf7a5832f006/volumes"
Oct 30 21:31:47.276714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.276676    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"]
Oct 30 21:31:47.277610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.277587    5886 cpu_manager.go:414] "containerMap: removing container" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy"
Oct 30 21:31:47.277683 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.277613    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy"
Oct 30 21:31:47.277825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.277812    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy"
Oct 30 21:31:47.277872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.277836    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy"
Oct 30 21:31:47.283575 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.283553    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:47.284697 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.284670    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0a790f-42eb-4a41-aae9-718a87ef805f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5\" (UID: \"3b0a790f-42eb-4a41-aae9-718a87ef805f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:47.284825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.284791    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkknt\" (UniqueName: \"kubernetes.io/projected/3b0a790f-42eb-4a41-aae9-718a87ef805f-kube-api-access-fkknt\") pod \"payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5\" (UID: \"3b0a790f-42eb-4a41-aae9-718a87ef805f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:47.299311 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.299281    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"]
Oct 30 21:31:47.306630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.305455    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-18a043\""
Oct 30 21:31:47.306630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.305632    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-18a043-dockercfg-lkf27\""
Oct 30 21:31:47.419178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.395738    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0a790f-42eb-4a41-aae9-718a87ef805f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5\" (UID: \"3b0a790f-42eb-4a41-aae9-718a87ef805f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:47.433185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.429386    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkknt\" (UniqueName: \"kubernetes.io/projected/3b0a790f-42eb-4a41-aae9-718a87ef805f-kube-api-access-fkknt\") pod \"payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5\" (UID: \"3b0a790f-42eb-4a41-aae9-718a87ef805f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:47.433185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.407989    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0a790f-42eb-4a41-aae9-718a87ef805f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5\" (UID: \"3b0a790f-42eb-4a41-aae9-718a87ef805f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:47.476189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.453232    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"]
Oct 30 21:31:47.476189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.454511    5886 cpu_manager.go:414] "containerMap: removing container" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy"
Oct 30 21:31:47.476189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.454532    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6add9d4-c0a6-4f56-9c06-bf7a5832f006" containerName="istio-proxy"
Oct 30 21:31:47.476189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.455943    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkknt\" (UniqueName: \"kubernetes.io/projected/3b0a790f-42eb-4a41-aae9-718a87ef805f-kube-api-access-fkknt\") pod \"payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5\" (UID: \"3b0a790f-42eb-4a41-aae9-718a87ef805f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:47.479462 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.479437    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:47.486486 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.486450    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"]
Oct 30 21:31:47.535179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.535073    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-18a043-7575bd895c-prbxx\" (UID: \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\") " pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:47.538460 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.538430    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr2hc\" (UniqueName: \"kubernetes.io/projected/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-kube-api-access-jr2hc\") pod \"payload-processing-e2e-api-a-18a043-7575bd895c-prbxx\" (UID: \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\") " pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:47.604240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.604205    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:47.640578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.640542    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-18a043-7575bd895c-prbxx\" (UID: \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\") " pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:47.640737 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.640598    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr2hc\" (UniqueName: \"kubernetes.io/projected/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-kube-api-access-jr2hc\") pod \"payload-processing-e2e-api-a-18a043-7575bd895c-prbxx\" (UID: \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\") " pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:47.641394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.641367    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-18a043-7575bd895c-prbxx\" (UID: \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\") " pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:47.659790 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.659757    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr2hc\" (UniqueName: \"kubernetes.io/projected/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-kube-api-access-jr2hc\") pod \"payload-processing-e2e-api-a-18a043-7575bd895c-prbxx\" (UID: \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\") " pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:47.810454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.810365    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:47.932765 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:47.932732    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"]
Oct 30 21:31:47.935389 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:31:47.935351    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0a790f_42eb_4a41_aae9_718a87ef805f.slice/crio-db18332e25ea8e3bfe2c33e4f28ffa26b227c6469039a70f788d103713904eef WatchSource:0}: Error finding container db18332e25ea8e3bfe2c33e4f28ffa26b227c6469039a70f788d103713904eef: Status 404 returned error can't find the container with id db18332e25ea8e3bfe2c33e4f28ffa26b227c6469039a70f788d103713904eef
Oct 30 21:31:48.129887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.129790    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"]
Oct 30 21:31:48.137064 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:31:48.137026    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953c1b4e_ad15_4fbb_a1bd_c9b5aa41f90c.slice/crio-02bdf76bc8788dd6b291036fa11956c4e531e68dc65f862bbe52b0def8d2bbea WatchSource:0}: Error finding container 02bdf76bc8788dd6b291036fa11956c4e531e68dc65f862bbe52b0def8d2bbea: Status 404 returned error can't find the container with id 02bdf76bc8788dd6b291036fa11956c4e531e68dc65f862bbe52b0def8d2bbea
Oct 30 21:31:48.801143 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.801105    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx" event={"ID":"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c","Type":"ContainerStarted","Data":"94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6"}
Oct 30 21:31:48.801633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.801187    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx" event={"ID":"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c","Type":"ContainerStarted","Data":"02bdf76bc8788dd6b291036fa11956c4e531e68dc65f862bbe52b0def8d2bbea"}
Oct 30 21:31:48.801633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.801324    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:48.803035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.803009    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5" event={"ID":"3b0a790f-42eb-4a41-aae9-718a87ef805f","Type":"ContainerStarted","Data":"01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c"}
Oct 30 21:31:48.803177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.803056    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5" event={"ID":"3b0a790f-42eb-4a41-aae9-718a87ef805f","Type":"ContainerStarted","Data":"db18332e25ea8e3bfe2c33e4f28ffa26b227c6469039a70f788d103713904eef"}
Oct 30 21:31:48.803177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.803083    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:48.839983 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.839938    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx" podStartSLOduration=1.8399250459999998 podStartE2EDuration="1.839925046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:48.839320799 +0000 UTC m=+1308.968572273" watchObservedRunningTime="2026-10-30 21:31:48.839925046 +0000 UTC m=+1308.969176502"
Oct 30 21:31:48.878170 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:48.878111    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5" podStartSLOduration=1.87809629 podStartE2EDuration="1.87809629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:48.876467715 +0000 UTC m=+1309.005719174" watchObservedRunningTime="2026-10-30 21:31:48.87809629 +0000 UTC m=+1309.007347750"
Oct 30 21:31:50.689696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:50.689662    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"]
Oct 30 21:31:50.723341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:50.723304    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"]
Oct 30 21:31:55.690719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:55.690652    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x" podUID="cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f" containerName="payload-processing" containerID="cri-o://95caf17b0d165343da3b1a2505ae26b317db849a920f8875159f6fbd98e374c7" gracePeriod=25
Oct 30 21:31:55.723983 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:55.723950    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt" podUID="d48c76be-d415-4a41-aeaf-6e7522cffe64" containerName="payload-pre-processing" containerID="cri-o://cd991e98e13a74221f12ef6ea0970bb1e1d228bb720403dbc81ddef8b55ea25d" gracePeriod=25
Oct 30 21:31:55.861329 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:55.861288    5886 generic.go:356] "Generic (PLEG): container finished" podID="cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f" containerID="95caf17b0d165343da3b1a2505ae26b317db849a920f8875159f6fbd98e374c7" exitCode=0
Oct 30 21:31:55.861497 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:55.861451    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x" event={"ID":"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f","Type":"ContainerDied","Data":"95caf17b0d165343da3b1a2505ae26b317db849a920f8875159f6fbd98e374c7"}
Oct 30 21:31:55.863816 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:55.863786    5886 generic.go:356] "Generic (PLEG): container finished" podID="d48c76be-d415-4a41-aeaf-6e7522cffe64" containerID="cd991e98e13a74221f12ef6ea0970bb1e1d228bb720403dbc81ddef8b55ea25d" exitCode=0
Oct 30 21:31:55.863982 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:55.863966    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt" event={"ID":"d48c76be-d415-4a41-aeaf-6e7522cffe64","Type":"ContainerDied","Data":"cd991e98e13a74221f12ef6ea0970bb1e1d228bb720403dbc81ddef8b55ea25d"}
Oct 30 21:31:56.114642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.114292    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:56.119348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.118298    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:56.286673 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.286580    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-kube-api-access-kvmx2\" (UniqueName: \"kubernetes.io/projected/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-kube-api-access-kvmx2\") pod \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\" (UID: \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\") "
Oct 30 21:31:56.286830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.286715    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48c76be-d415-4a41-aeaf-6e7522cffe64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48c76be-d415-4a41-aeaf-6e7522cffe64-plugins-config-volume\") pod \"d48c76be-d415-4a41-aeaf-6e7522cffe64\" (UID: \"d48c76be-d415-4a41-aeaf-6e7522cffe64\") "
Oct 30 21:31:56.286830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.286775    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-plugins-config-volume\") pod \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\" (UID: \"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f\") "
Oct 30 21:31:56.286953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.286885    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48c76be-d415-4a41-aeaf-6e7522cffe64-kube-api-access-vwnjz\" (UniqueName: \"kubernetes.io/projected/d48c76be-d415-4a41-aeaf-6e7522cffe64-kube-api-access-vwnjz\") pod \"d48c76be-d415-4a41-aeaf-6e7522cffe64\" (UID: \"d48c76be-d415-4a41-aeaf-6e7522cffe64\") "
Oct 30 21:31:56.287175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.287127    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48c76be-d415-4a41-aeaf-6e7522cffe64-plugins-config-volume" pod "d48c76be-d415-4a41-aeaf-6e7522cffe64" (UID: "d48c76be-d415-4a41-aeaf-6e7522cffe64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:56.287364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.287208    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-plugins-config-volume" pod "cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f" (UID: "cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:56.287527 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.287507    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d48c76be-d415-4a41-aeaf-6e7522cffe64-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:56.287605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.287532    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:56.288957 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.288935    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48c76be-d415-4a41-aeaf-6e7522cffe64-kube-api-access-vwnjz" pod "d48c76be-d415-4a41-aeaf-6e7522cffe64" (UID: "d48c76be-d415-4a41-aeaf-6e7522cffe64"). InnerVolumeSpecName "kube-api-access-vwnjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:56.289192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.289168    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-kube-api-access-kvmx2" pod "cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f" (UID: "cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f"). InnerVolumeSpecName "kube-api-access-kvmx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:56.388655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.388627    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvmx2\" (UniqueName: \"kubernetes.io/projected/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f-kube-api-access-kvmx2\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:56.388655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.388657    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwnjz\" (UniqueName: \"kubernetes.io/projected/d48c76be-d415-4a41-aeaf-6e7522cffe64-kube-api-access-vwnjz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:31:56.873442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.873410    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"
Oct 30 21:31:56.873442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.873438    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x" event={"ID":"cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f","Type":"ContainerDied","Data":"d95e0de856e7c19c43e59e4e1a472eca92e82f56e44dec584789a9b8a372ff04"}
Oct 30 21:31:56.873993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.873500    5886 scope.go:122] "RemoveContainer" containerID="95caf17b0d165343da3b1a2505ae26b317db849a920f8875159f6fbd98e374c7"
Oct 30 21:31:56.875431 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.875411    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"
Oct 30 21:31:56.875553 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.875437    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt" event={"ID":"d48c76be-d415-4a41-aeaf-6e7522cffe64","Type":"ContainerDied","Data":"27e91a0ece67453c610fd712bcb1629487802c8144b103f33bc1bccd2e63f9e4"}
Oct 30 21:31:56.887880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.887859    5886 scope.go:122] "RemoveContainer" containerID="cd991e98e13a74221f12ef6ea0970bb1e1d228bb720403dbc81ddef8b55ea25d"
Oct 30 21:31:56.924966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.924922    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"]
Oct 30 21:31:56.935188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.935136    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-6c8857b547-2l4qt"]
Oct 30 21:31:56.961010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.960948    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"]
Oct 30 21:31:56.965695 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:56.965668    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c027ae-5cc8bb88bf-x2c5x"]
Oct 30 21:31:57.916225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.916194    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"]
Oct 30 21:31:57.916989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.916965    5886 cpu_manager.go:414] "containerMap: removing container" podUID="cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f" containerName="payload-processing"
Oct 30 21:31:57.917096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.916990    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f" containerName="payload-processing"
Oct 30 21:31:57.917096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.917038    5886 cpu_manager.go:414] "containerMap: removing container" podUID="d48c76be-d415-4a41-aeaf-6e7522cffe64" containerName="payload-pre-processing"
Oct 30 21:31:57.917096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.917047    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48c76be-d415-4a41-aeaf-6e7522cffe64" containerName="payload-pre-processing"
Oct 30 21:31:57.917269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.917196    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48c76be-d415-4a41-aeaf-6e7522cffe64" containerName="payload-pre-processing"
Oct 30 21:31:57.917269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.917211    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f" containerName="payload-processing"
Oct 30 21:31:57.923431 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.923397    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:31:57.939205 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.939180    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"]
Oct 30 21:31:57.945840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.945806    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-229795\""
Oct 30 21:31:57.945953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:57.945873    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-229795-dockercfg-pftfz\""
Oct 30 21:31:58.014178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.012982    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f" path="/var/lib/kubelet/pods/cd5dc2f5-15ce-4cf5-9a5e-0fab34e3518f/volumes"
Oct 30 21:31:58.014178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.013760    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48c76be-d415-4a41-aeaf-6e7522cffe64" path="/var/lib/kubelet/pods/d48c76be-d415-4a41-aeaf-6e7522cffe64/volumes"
Oct 30 21:31:58.015300 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.015271    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkwjk\" (UniqueName: \"kubernetes.io/projected/a18994f6-014f-4afd-8f50-fc672581c6ce-kube-api-access-dkwjk\") pod \"payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5\" (UID: \"a18994f6-014f-4afd-8f50-fc672581c6ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:31:58.015501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.015470    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18994f6-014f-4afd-8f50-fc672581c6ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5\" (UID: \"a18994f6-014f-4afd-8f50-fc672581c6ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:31:58.117026 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.116984    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18994f6-014f-4afd-8f50-fc672581c6ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5\" (UID: \"a18994f6-014f-4afd-8f50-fc672581c6ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:31:58.117243 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.117231    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwjk\" (UniqueName: \"kubernetes.io/projected/a18994f6-014f-4afd-8f50-fc672581c6ce-kube-api-access-dkwjk\") pod \"payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5\" (UID: \"a18994f6-014f-4afd-8f50-fc672581c6ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:31:58.117687 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.117665    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18994f6-014f-4afd-8f50-fc672581c6ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5\" (UID: \"a18994f6-014f-4afd-8f50-fc672581c6ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:31:58.136922 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.136743    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwjk\" (UniqueName: \"kubernetes.io/projected/a18994f6-014f-4afd-8f50-fc672581c6ce-kube-api-access-dkwjk\") pod \"payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5\" (UID: \"a18994f6-014f-4afd-8f50-fc672581c6ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:31:58.241425 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.241327    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:31:58.372572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.372545    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"]
Oct 30 21:31:58.378647 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.378620    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:31:58.403962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.403922    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"]
Oct 30 21:31:58.424601 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.424560    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd8e513-5ee6-46a9-84f7-1ef53499be33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9\" (UID: \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:31:58.424766 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.424734    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc76r\" (UniqueName: \"kubernetes.io/projected/4cd8e513-5ee6-46a9-84f7-1ef53499be33-kube-api-access-lc76r\") pod \"payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9\" (UID: \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:31:58.526587 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.526483    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc76r\" (UniqueName: \"kubernetes.io/projected/4cd8e513-5ee6-46a9-84f7-1ef53499be33-kube-api-access-lc76r\") pod \"payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9\" (UID: \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:31:58.526794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.526773    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd8e513-5ee6-46a9-84f7-1ef53499be33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9\" (UID: \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:31:58.527494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.527473    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd8e513-5ee6-46a9-84f7-1ef53499be33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9\" (UID: \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:31:58.555658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.555627    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc76r\" (UniqueName: \"kubernetes.io/projected/4cd8e513-5ee6-46a9-84f7-1ef53499be33-kube-api-access-lc76r\") pod \"payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9\" (UID: \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:31:58.599175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.598578    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"]
Oct 30 21:31:58.612051 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:31:58.612016    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18994f6_014f_4afd_8f50_fc672581c6ce.slice/crio-ccb231939a4e8abc9125bb63637b5de0d63dd554f096f127b98ee4d65a0286ec WatchSource:0}: Error finding container ccb231939a4e8abc9125bb63637b5de0d63dd554f096f127b98ee4d65a0286ec: Status 404 returned error can't find the container with id ccb231939a4e8abc9125bb63637b5de0d63dd554f096f127b98ee4d65a0286ec
Oct 30 21:31:58.708888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.708857    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:31:58.896295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:58.896207    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5" event={"ID":"a18994f6-014f-4afd-8f50-fc672581c6ce","Type":"ContainerStarted","Data":"ccb231939a4e8abc9125bb63637b5de0d63dd554f096f127b98ee4d65a0286ec"}
Oct 30 21:31:59.018553 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:59.018529    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"]
Oct 30 21:31:59.021069 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:31:59.021045    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd8e513_5ee6_46a9_84f7_1ef53499be33.slice/crio-34f42a9a8d3a226dd3dd8c1ca44afe9b38d6bed25423347791256ea3348018fb WatchSource:0}: Error finding container 34f42a9a8d3a226dd3dd8c1ca44afe9b38d6bed25423347791256ea3348018fb: Status 404 returned error can't find the container with id 34f42a9a8d3a226dd3dd8c1ca44afe9b38d6bed25423347791256ea3348018fb
Oct 30 21:31:59.816513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:59.816234    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:31:59.816737 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:59.816710    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:31:59.925949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:31:59.913569    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9" event={"ID":"4cd8e513-5ee6-46a9-84f7-1ef53499be33","Type":"ContainerStarted","Data":"34f42a9a8d3a226dd3dd8c1ca44afe9b38d6bed25423347791256ea3348018fb"}
Oct 30 21:32:00.930092 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:00.930044    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9" event={"ID":"4cd8e513-5ee6-46a9-84f7-1ef53499be33","Type":"ContainerStarted","Data":"db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89"}
Oct 30 21:32:00.930787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:00.930755    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:32:00.932952 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:00.932919    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5" event={"ID":"a18994f6-014f-4afd-8f50-fc672581c6ce","Type":"ContainerStarted","Data":"8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51"}
Oct 30 21:32:00.933831 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:00.933805    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:32:00.974066 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:00.974007    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9" podStartSLOduration=1.684052522 podStartE2EDuration="2.973993749s" totalImagesPullingTime="1.289941227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:00.969327275 +0000 UTC m=+1321.098578732" watchObservedRunningTime="2026-10-30 21:32:00.973993749 +0000 UTC m=+1321.103245229"
Oct 30 21:32:01.012000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.011950    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5" podStartSLOduration=2.373821109 podStartE2EDuration="4.01193617s" totalImagesPullingTime="1.638115061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:01.006114839 +0000 UTC m=+1321.135366297" watchObservedRunningTime="2026-10-30 21:32:01.01193617 +0000 UTC m=+1321.141187627"
Oct 30 21:32:01.400747 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.400704    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"]
Oct 30 21:32:01.408749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.408721    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.427958 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.427903    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-18a043-serving-cert\""
Oct 30 21:32:01.429703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.429586    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"]
Oct 30 21:32:01.537175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.534362    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-openshift-service-ca\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.537175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.534526    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-metrics-certs\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.537175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.534639    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgqmt\" (UniqueName: \"kubernetes.io/projected/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-kube-api-access-kgqmt\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.537175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.534696    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-maas-api-tls\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.636435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.636130    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-metrics-certs\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.636435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.636253    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgqmt\" (UniqueName: \"kubernetes.io/projected/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-kube-api-access-kgqmt\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.636435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.636300    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-maas-api-tls\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.636435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.636387    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-openshift-service-ca\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.637477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.637450    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-openshift-service-ca\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.639032 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.639007    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-metrics-certs\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.639196 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.639141    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-maas-api-tls\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.655764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.655735    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgqmt\" (UniqueName: \"kubernetes.io/projected/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-kube-api-access-kgqmt\") pod \"maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:01.729180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:01.728495    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:02.082418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:02.082383    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"]
Oct 30 21:32:02.087890 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:32:02.087853    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3df7fb_98d8_4ea6_8e16_0a398d776d64.slice/crio-522ac56cce6ee02b76cd4b0ef5e325525efc33e13cf0384d68d0902aef05fcea WatchSource:0}: Error finding container 522ac56cce6ee02b76cd4b0ef5e325525efc33e13cf0384d68d0902aef05fcea: Status 404 returned error can't find the container with id 522ac56cce6ee02b76cd4b0ef5e325525efc33e13cf0384d68d0902aef05fcea
Oct 30 21:32:02.973117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:02.973053    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n" event={"ID":"aa3df7fb-98d8-4ea6-8e16-0a398d776d64","Type":"ContainerStarted","Data":"3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194"}
Oct 30 21:32:02.973337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:02.973141    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n" event={"ID":"aa3df7fb-98d8-4ea6-8e16-0a398d776d64","Type":"ContainerStarted","Data":"522ac56cce6ee02b76cd4b0ef5e325525efc33e13cf0384d68d0902aef05fcea"}
Oct 30 21:32:02.973337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:02.973206    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:03.020442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:03.020383    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n" podStartSLOduration=1.697519303 podStartE2EDuration="2.020368147s" totalImagesPullingTime="322.848844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:03.016466958 +0000 UTC m=+1323.145718415" watchObservedRunningTime="2026-10-30 21:32:03.020368147 +0000 UTC m=+1323.149619636"
Oct 30 21:32:06.100899 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.100856    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"]
Oct 30 21:32:06.105723 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.105700    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:06.119962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.119938    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb23ae2-c8c6-4dce-ad29-682697b0851b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7\" (UID: \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:06.120085 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.119966    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhklk\" (UniqueName: \"kubernetes.io/projected/6fb23ae2-c8c6-4dce-ad29-682697b0851b-kube-api-access-vhklk\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7\" (UID: \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:06.125966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.125939    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"]
Oct 30 21:32:06.199228 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.199183    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c027ae\""
Oct 30 21:32:06.199393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.199361    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c027ae-dockercfg-hwn2d\""
Oct 30 21:32:06.221339 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.221307    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb23ae2-c8c6-4dce-ad29-682697b0851b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7\" (UID: \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:06.221480 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.221343    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhklk\" (UniqueName: \"kubernetes.io/projected/6fb23ae2-c8c6-4dce-ad29-682697b0851b-kube-api-access-vhklk\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7\" (UID: \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:06.221977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.221957    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb23ae2-c8c6-4dce-ad29-682697b0851b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7\" (UID: \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:06.239656 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.239622    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhklk\" (UniqueName: \"kubernetes.io/projected/6fb23ae2-c8c6-4dce-ad29-682697b0851b-kube-api-access-vhklk\") pod \"payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7\" (UID: \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:06.421806 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.421778    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:06.724000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:06.723975    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"]
Oct 30 21:32:06.726398 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:32:06.726371    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb23ae2_c8c6_4dce_ad29_682697b0851b.slice/crio-a7280cb555695c04f2f6f2e71dcd0635524f7a9c0ea58e0d78ce29dc6b01eea0 WatchSource:0}: Error finding container a7280cb555695c04f2f6f2e71dcd0635524f7a9c0ea58e0d78ce29dc6b01eea0: Status 404 returned error can't find the container with id a7280cb555695c04f2f6f2e71dcd0635524f7a9c0ea58e0d78ce29dc6b01eea0
Oct 30 21:32:07.012455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:07.010553    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7" event={"ID":"6fb23ae2-c8c6-4dce-ad29-682697b0851b","Type":"ContainerStarted","Data":"687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c"}
Oct 30 21:32:07.012455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:07.010619    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7" event={"ID":"6fb23ae2-c8c6-4dce-ad29-682697b0851b","Type":"ContainerStarted","Data":"a7280cb555695c04f2f6f2e71dcd0635524f7a9c0ea58e0d78ce29dc6b01eea0"}
Oct 30 21:32:07.012455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:07.012208    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:07.052079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:07.052023    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7" podStartSLOduration=1.052008182 podStartE2EDuration="1.052008182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:07.050023457 +0000 UTC m=+1327.179274914" watchObservedRunningTime="2026-10-30 21:32:07.052008182 +0000 UTC m=+1327.181259639"
Oct 30 21:32:08.828705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:08.828672    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"]
Oct 30 21:32:08.833084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:08.833063    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:08.851744 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:08.851716    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0a1ee3\""
Oct 30 21:32:08.862923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:08.862886    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"]
Oct 30 21:32:08.866045 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:08.866011    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgmp\" (UniqueName: \"kubernetes.io/projected/b25778ff-68d1-41dc-ba31-d64258679d40-kube-api-access-kwgmp\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf\" (UID: \"b25778ff-68d1-41dc-ba31-d64258679d40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:08.866223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:08.866144    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25778ff-68d1-41dc-ba31-d64258679d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf\" (UID: \"b25778ff-68d1-41dc-ba31-d64258679d40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:08.885644 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:08.885616    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0a1ee3-dockercfg-g7hml\""
Oct 30 21:32:09.003583 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:08.969607    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgmp\" (UniqueName: \"kubernetes.io/projected/b25778ff-68d1-41dc-ba31-d64258679d40-kube-api-access-kwgmp\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf\" (UID: \"b25778ff-68d1-41dc-ba31-d64258679d40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:09.003961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:09.003937    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25778ff-68d1-41dc-ba31-d64258679d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf\" (UID: \"b25778ff-68d1-41dc-ba31-d64258679d40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:09.012594 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:09.004942    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25778ff-68d1-41dc-ba31-d64258679d40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf\" (UID: \"b25778ff-68d1-41dc-ba31-d64258679d40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:09.047169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:09.047118    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:32:09.064376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:09.064339    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgmp\" (UniqueName: \"kubernetes.io/projected/b25778ff-68d1-41dc-ba31-d64258679d40-kube-api-access-kwgmp\") pod \"payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf\" (UID: \"b25778ff-68d1-41dc-ba31-d64258679d40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:09.148430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:09.148387    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:09.487372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:09.487335    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"]
Oct 30 21:32:09.491199 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:32:09.489594    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25778ff_68d1_41dc_ba31_d64258679d40.slice/crio-654feb566de1fe5d44f12971f65b89ab33c48c898119e2f2a3e35d728e3661a6 WatchSource:0}: Error finding container 654feb566de1fe5d44f12971f65b89ab33c48c898119e2f2a3e35d728e3661a6: Status 404 returned error can't find the container with id 654feb566de1fe5d44f12971f65b89ab33c48c898119e2f2a3e35d728e3661a6
Oct 30 21:32:10.097027 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:10.096990    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf" event={"ID":"b25778ff-68d1-41dc-ba31-d64258679d40","Type":"ContainerStarted","Data":"fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886"}
Oct 30 21:32:10.097461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:10.097043    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf" event={"ID":"b25778ff-68d1-41dc-ba31-d64258679d40","Type":"ContainerStarted","Data":"654feb566de1fe5d44f12971f65b89ab33c48c898119e2f2a3e35d728e3661a6"}
Oct 30 21:32:10.097461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:10.097073    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:10.140362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:10.140299    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf" podStartSLOduration=2.140281638 podStartE2EDuration="2.140281638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:10.136368242 +0000 UTC m=+1330.265619699" watchObservedRunningTime="2026-10-30 21:32:10.140281638 +0000 UTC m=+1330.269533099"
Oct 30 21:32:11.032808 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:32:11.032769    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:32:11.965939 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:11.965902    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:32:12.976306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:12.976277    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:32:17.513541 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.513504    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"]
Oct 30 21:32:17.518558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.518486    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.534586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.534556    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3afc5e-serving-cert\""
Oct 30 21:32:17.547234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.547204    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c26q\" (UniqueName: \"kubernetes.io/projected/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-kube-api-access-8c26q\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.547464 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.547428    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.547785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.547762    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-metrics-certs\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.547860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.547816    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.551517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.551491    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"]
Oct 30 21:32:17.649515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.649480    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c26q\" (UniqueName: \"kubernetes.io/projected/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-kube-api-access-8c26q\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.649697 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.649524    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.649777 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.649695    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-metrics-certs\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.649777 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.649735    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.650444 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.650417    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.652018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.651995    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.652233 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.652216    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-metrics-certs\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.668703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.668672    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c26q\" (UniqueName: \"kubernetes.io/projected/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-kube-api-access-8c26q\") pod \"maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:17.836971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:17.836866    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:18.184179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:18.171745    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"]
Oct 30 21:32:18.184179 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:32:18.177310    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e4e2c3_1cc7_4caf_bbfa_bf4f365b4126.slice/crio-16036c954921d7da928cb71fec2bac043d49cacd62f3a0cc66e307ed53594a06 WatchSource:0}: Error finding container 16036c954921d7da928cb71fec2bac043d49cacd62f3a0cc66e307ed53594a06: Status 404 returned error can't find the container with id 16036c954921d7da928cb71fec2bac043d49cacd62f3a0cc66e307ed53594a06
Oct 30 21:32:19.073984 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:19.073953    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:32:19.171364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:19.171325    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn" event={"ID":"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126","Type":"ContainerStarted","Data":"3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734"}
Oct 30 21:32:19.171535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:19.171387    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn" event={"ID":"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126","Type":"ContainerStarted","Data":"16036c954921d7da928cb71fec2bac043d49cacd62f3a0cc66e307ed53594a06"}
Oct 30 21:32:19.171535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:19.171413    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:19.212557 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:19.212496    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn" podStartSLOduration=1.827699128 podStartE2EDuration="2.212480131s" totalImagesPullingTime="384.781003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:19.210255368 +0000 UTC m=+1339.339506827" watchObservedRunningTime="2026-10-30 21:32:19.212480131 +0000 UTC m=+1339.341731590"
Oct 30 21:32:20.961245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:20.961212    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"]
Oct 30 21:32:20.967842 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:20.967815    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:20.983048 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:20.982864    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7222e2-dockercfg-tm2sx\""
Oct 30 21:32:20.983048 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:20.982962    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7222e2\""
Oct 30 21:32:20.994536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:20.994505    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"]
Oct 30 21:32:21.107438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.107364    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:32:21.131733 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.131698    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb\" (UID: \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:21.131919 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.131866    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rkq\" (UniqueName: \"kubernetes.io/projected/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-kube-api-access-z4rkq\") pod \"payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb\" (UID: \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:21.233512 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.233433    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb\" (UID: \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:21.233671 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.233559    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rkq\" (UniqueName: \"kubernetes.io/projected/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-kube-api-access-z4rkq\") pod \"payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb\" (UID: \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:21.234238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.234218    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb\" (UID: \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:21.253185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.253131    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rkq\" (UniqueName: \"kubernetes.io/projected/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-kube-api-access-z4rkq\") pod \"payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb\" (UID: \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:21.293665 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.293628    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:21.613348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:21.613317    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"]
Oct 30 21:32:21.615630 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:32:21.615599    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fdc49f_ebc3_497a_bb3c_39e71ca3da5a.slice/crio-b0b5ae1ea1b7683170fe00ceaedc6cf661e8abffba79ad4672fdc7d8df9dd21a WatchSource:0}: Error finding container b0b5ae1ea1b7683170fe00ceaedc6cf661e8abffba79ad4672fdc7d8df9dd21a: Status 404 returned error can't find the container with id b0b5ae1ea1b7683170fe00ceaedc6cf661e8abffba79ad4672fdc7d8df9dd21a
Oct 30 21:32:22.200307 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:22.200262    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb" event={"ID":"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a","Type":"ContainerStarted","Data":"6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d"}
Oct 30 21:32:22.200838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:22.200344    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb" event={"ID":"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a","Type":"ContainerStarted","Data":"b0b5ae1ea1b7683170fe00ceaedc6cf661e8abffba79ad4672fdc7d8df9dd21a"}
Oct 30 21:32:22.200838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:22.200518    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:22.240503 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:22.240454    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb" podStartSLOduration=2.2404397879999998 podStartE2EDuration="2.240439788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:22.239247937 +0000 UTC m=+1342.368499393" watchObservedRunningTime="2026-10-30 21:32:22.240439788 +0000 UTC m=+1342.369691277"
Oct 30 21:32:25.186282 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:25.186250    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:32:30.208088 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.208051    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"]
Oct 30 21:32:30.214571 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.214552    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:30.231931 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.231233    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"]
Oct 30 21:32:30.237299 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.237274    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e38f60\""
Oct 30 21:32:30.237413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.237324    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e38f60-dockercfg-52kpc\""
Oct 30 21:32:30.406194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.396892    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc\" (UID: \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:30.406194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.397301    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rns\" (UniqueName: \"kubernetes.io/projected/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-kube-api-access-w6rns\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc\" (UID: \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:30.514629 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.514535    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rns\" (UniqueName: \"kubernetes.io/projected/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-kube-api-access-w6rns\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc\" (UID: \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:30.514944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.514919    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc\" (UID: \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:30.516252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.515927    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc\" (UID: \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:30.543889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.543855    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rns\" (UniqueName: \"kubernetes.io/projected/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-kube-api-access-w6rns\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc\" (UID: \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:30.556350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.556322    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"]
Oct 30 21:32:30.561098 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.561075    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"]
Oct 30 21:32:30.561256 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.561184    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:30.719615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.719582    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff30c120-098b-45d9-8dd5-0cc84e53e308-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc\" (UID: \"ff30c120-098b-45d9-8dd5-0cc84e53e308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:30.719615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.719618    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx462\" (UniqueName: \"kubernetes.io/projected/ff30c120-098b-45d9-8dd5-0cc84e53e308-kube-api-access-rx462\") pod \"payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc\" (UID: \"ff30c120-098b-45d9-8dd5-0cc84e53e308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:30.820959 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.820869    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff30c120-098b-45d9-8dd5-0cc84e53e308-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc\" (UID: \"ff30c120-098b-45d9-8dd5-0cc84e53e308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:30.820959 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.820913    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx462\" (UniqueName: \"kubernetes.io/projected/ff30c120-098b-45d9-8dd5-0cc84e53e308-kube-api-access-rx462\") pod \"payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc\" (UID: \"ff30c120-098b-45d9-8dd5-0cc84e53e308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:30.822043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.821994    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff30c120-098b-45d9-8dd5-0cc84e53e308-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc\" (UID: \"ff30c120-098b-45d9-8dd5-0cc84e53e308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:30.831630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.831603    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:30.841413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.841386    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx462\" (UniqueName: \"kubernetes.io/projected/ff30c120-098b-45d9-8dd5-0cc84e53e308-kube-api-access-rx462\") pod \"payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc\" (UID: \"ff30c120-098b-45d9-8dd5-0cc84e53e308\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:30.916033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:30.915993    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:31.190250 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.189789    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"]
Oct 30 21:32:31.284068 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:32:31.284030    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff30c120_098b_45d9_8dd5_0cc84e53e308.slice/crio-ac3640df9ec6154c55ad18a777f6a12a78803293f85ddc677f34d3699144091c WatchSource:0}: Error finding container ac3640df9ec6154c55ad18a777f6a12a78803293f85ddc677f34d3699144091c: Status 404 returned error can't find the container with id ac3640df9ec6154c55ad18a777f6a12a78803293f85ddc677f34d3699144091c
Oct 30 21:32:31.285457 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.285436    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"]
Oct 30 21:32:31.554809 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.554725    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc" event={"ID":"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc","Type":"ContainerStarted","Data":"8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027"}
Oct 30 21:32:31.554809 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.554806    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:31.555040 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.554828    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc" event={"ID":"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc","Type":"ContainerStarted","Data":"ca5df42c59dd972e238fe9d1c5e0d6076efa78bbddd85d44c540f042378d4f5a"}
Oct 30 21:32:31.556723 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.556679    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc" event={"ID":"ff30c120-098b-45d9-8dd5-0cc84e53e308","Type":"ContainerStarted","Data":"7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e"}
Oct 30 21:32:31.556880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.556739    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc" event={"ID":"ff30c120-098b-45d9-8dd5-0cc84e53e308","Type":"ContainerStarted","Data":"ac3640df9ec6154c55ad18a777f6a12a78803293f85ddc677f34d3699144091c"}
Oct 30 21:32:31.556880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.556849    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:31.598789 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.598712    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc" podStartSLOduration=1.598693435 podStartE2EDuration="1.598693435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:31.594090721 +0000 UTC m=+1351.723342180" watchObservedRunningTime="2026-10-30 21:32:31.598693435 +0000 UTC m=+1351.727944896"
Oct 30 21:32:31.633865 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:31.633811    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc" podStartSLOduration=1.633797629 podStartE2EDuration="1.633797629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:31.630822627 +0000 UTC m=+1351.760074083" watchObservedRunningTime="2026-10-30 21:32:31.633797629 +0000 UTC m=+1351.763049114"
Oct 30 21:32:33.220444 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:33.220408    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:32:35.592568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:35.592521    5886 generic.go:356] "Generic (PLEG): container finished" podID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerID="8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9" exitCode=0
Oct 30 21:32:35.593033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:35.592598    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" event={"ID":"9fcc8030-8b61-4bf2-aff3-423a4cf469f7","Type":"ContainerDied","Data":"8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9"}
Oct 30 21:32:35.593106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:35.593055    5886 scope.go:122] "RemoveContainer" containerID="8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9"
Oct 30 21:32:35.594715 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:35.594688    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:35.594802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:35.594765    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:35.594841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:35.594800    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:32:36.602467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:36.602429    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" event={"ID":"9fcc8030-8b61-4bf2-aff3-423a4cf469f7","Type":"ContainerStarted","Data":"3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65"}
Oct 30 21:32:37.041092 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:37.041053    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:32:37.042622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:37.042595    5886 patch_prober.go:29] interesting pod/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:32:37.042757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:37.042654    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:32:38.041527 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:38.041488    5886 patch_prober.go:29] interesting pod/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:32:38.042012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:38.041557    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:32:39.040891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:39.040858    5886 patch_prober.go:29] interesting pod/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:32:39.041084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:39.040914    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:32:40.041242 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:40.041204    5886 patch_prober.go:29] interesting pod/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:32:40.041627 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:40.041258    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:32:41.031277 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:32:41.031199    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:32:41.041165 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.041129    5886 patch_prober.go:29] interesting pod/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:32:41.041290 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.041190    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:32:41.488179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.487501    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"]
Oct 30 21:32:41.488179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.487756    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy" containerID="cri-o://f600f84467eacb671c4e93d525adb96b913947a1f0a4bb9f72bc65b907546117" gracePeriod=30
Oct 30 21:32:41.515543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.514322    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"]
Oct 30 21:32:41.515543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.514576    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy" containerID="cri-o://351ff491b0b6149e0634c16e59a92aa4545ed1d52701b52105d19de896018759" gracePeriod=30
Oct 30 21:32:41.522818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.522783    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"]
Oct 30 21:32:41.523297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.523272    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy" containerID="cri-o://c30f61be4b801a21c83e943cbd8b0a4752db3f6582aeac77720ccb74933c0d96" gracePeriod=30
Oct 30 21:32:41.526684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.526664    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"]
Oct 30 21:32:41.526887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:41.526865    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy" containerID="cri-o://eea3e934e803696f2b9c252c28eee4c3d1e989b13773aff45624a926138f71d4" gracePeriod=30
Oct 30 21:32:42.045117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:42.045089    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:32:42.045551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:42.045380    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:32:42.046197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:42.046144    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:32:42.568834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:42.568795    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:32:42.569037 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:42.568915    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:32:46.714208 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:46.713090    5886 generic.go:356] "Generic (PLEG): container finished" podID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerID="eea3e934e803696f2b9c252c28eee4c3d1e989b13773aff45624a926138f71d4" exitCode=0
Oct 30 21:32:46.714208 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:46.713335    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" event={"ID":"652750b3-8ca9-481a-b675-1ee10b1b3a2e","Type":"ContainerDied","Data":"eea3e934e803696f2b9c252c28eee4c3d1e989b13773aff45624a926138f71d4"}
Oct 30 21:32:46.716798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:46.716645    5886 generic.go:356] "Generic (PLEG): container finished" podID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerID="351ff491b0b6149e0634c16e59a92aa4545ed1d52701b52105d19de896018759" exitCode=0
Oct 30 21:32:46.716798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:46.716726    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" event={"ID":"59caa8f3-3d24-4f7a-8d4f-78914d2cef09","Type":"ContainerDied","Data":"351ff491b0b6149e0634c16e59a92aa4545ed1d52701b52105d19de896018759"}
Oct 30 21:32:46.719241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:46.719072    5886 generic.go:356] "Generic (PLEG): container finished" podID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerID="c30f61be4b801a21c83e943cbd8b0a4752db3f6582aeac77720ccb74933c0d96" exitCode=0
Oct 30 21:32:46.719241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:46.719171    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" event={"ID":"7959c1d6-8d4c-478f-86e6-e4795aa0f110","Type":"ContainerDied","Data":"c30f61be4b801a21c83e943cbd8b0a4752db3f6582aeac77720ccb74933c0d96"}
Oct 30 21:32:46.721942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:46.721872    5886 generic.go:356] "Generic (PLEG): container finished" podID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerID="f600f84467eacb671c4e93d525adb96b913947a1f0a4bb9f72bc65b907546117" exitCode=0
Oct 30 21:32:46.722051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:46.721965    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" event={"ID":"432e6150-2d0b-4e0a-9514-08e0fec591d7","Type":"ContainerDied","Data":"f600f84467eacb671c4e93d525adb96b913947a1f0a4bb9f72bc65b907546117"}
Oct 30 21:32:47.004633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.004591    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:32:47.037285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.037259    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:32:47.042941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.042922    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:32:47.053682 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.053662    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:32:47.133640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133604    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istiod-ca-cert\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.133640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133666    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-data\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.133949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133699    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-data\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.133949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133729    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-token\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-token\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.133949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133764    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-podinfo\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.133949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133812    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-kube-api-access-b6k9m\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.133949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133839    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-credential-socket\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.133949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133891    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-kube-api-access-k9spz\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-kube-api-access-k9spz\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.134321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133950    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-token\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-token\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.134321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.133981    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-kube-api-access-nb2tr\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-kube-api-access-nb2tr\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.134321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134009    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-data\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.134321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134041    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-podinfo\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.134321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134070    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-socket\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.134321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134116    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-podinfo\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.134321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134207    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-certs\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.134321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134244    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-certs\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.134738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134397    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-podinfo\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.134738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134431    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-credential-socket\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.134738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134473    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432e6150-2d0b-4e0a-9514-08e0fec591d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432e6150-2d0b-4e0a-9514-08e0fec591d7-istiod-ca-cert\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.134738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134511    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-socket\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.134738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134546    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-socket\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.134738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134607    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-envoy\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.134738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134691    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-envoy\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.134738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134721    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-kube-api-access-5dw27\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-kube-api-access-5dw27\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134767    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-socket\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134806    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-credential-socket\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134856    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-data\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134902    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istiod-ca-cert\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134942    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-credential-socket\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.134972    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-token\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-token\") pod \"432e6150-2d0b-4e0a-9514-08e0fec591d7\" (UID: \"432e6150-2d0b-4e0a-9514-08e0fec591d7\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.135019    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-envoy\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.135100    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-envoy\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.135129    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-certs\") pod \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\" (UID: \"59caa8f3-3d24-4f7a-8d4f-78914d2cef09\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.135199    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istiod-ca-cert\") pod \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\" (UID: \"7959c1d6-8d4c-478f-86e6-e4795aa0f110\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.135264    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-token\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-token\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.135369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.135343    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-certs\") pod \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\" (UID: \"652750b3-8ca9-481a-b675-1ee10b1b3a2e\") "
Oct 30 21:32:47.136371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.136313    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-data" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.136371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.136354    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-certs" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.136628 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.136602    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-credential-socket" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.136916 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.136890    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432e6150-2d0b-4e0a-9514-08e0fec591d7-istiod-ca-cert" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:47.137341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.137315    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istiod-ca-cert" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:47.138211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.138150    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-credential-socket" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.140251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.140221    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-certs" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.140374 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.140348    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-podinfo" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:47.140680 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.140540    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istiod-ca-cert" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:47.140768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.140747    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-data" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.141125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.141100    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-token" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:47.141287 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.141207    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-envoy" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.141684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.141661    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istiod-ca-cert" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:47.142043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.142020    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-data" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.142361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.142341    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-data" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.145851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.145410    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-socket" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.145851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.145622    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-socket" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.145851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.145834    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-token" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:47.147379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.147129    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-podinfo" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:47.147379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.147348    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-kube-api-access-5dw27" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "kube-api-access-5dw27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:47.147579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.147552    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-socket" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.147579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.147567    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-socket" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.148038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.147746    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-credential-socket" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.148038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.147760    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-certs" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.148038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.147946    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-certs" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.148241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.148131    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-credential-socket" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.149708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.149397    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-envoy" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.149708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.149465    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-envoy" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.151282 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.150626    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-kube-api-access-k9spz" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "kube-api-access-k9spz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:47.151282 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.150754    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-envoy" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:47.152203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.152143    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-token" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:47.156624 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.154403    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-podinfo" pod "7959c1d6-8d4c-478f-86e6-e4795aa0f110" (UID: "7959c1d6-8d4c-478f-86e6-e4795aa0f110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:47.156799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.156767    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-podinfo" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:47.156976 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.156931    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-kube-api-access-nb2tr" pod "432e6150-2d0b-4e0a-9514-08e0fec591d7" (UID: "432e6150-2d0b-4e0a-9514-08e0fec591d7"). InnerVolumeSpecName "kube-api-access-nb2tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:47.156976 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.156965    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-token" pod "59caa8f3-3d24-4f7a-8d4f-78914d2cef09" (UID: "59caa8f3-3d24-4f7a-8d4f-78914d2cef09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:47.158668 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.158647    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-kube-api-access-b6k9m" pod "652750b3-8ca9-481a-b675-1ee10b1b3a2e" (UID: "652750b3-8ca9-481a-b675-1ee10b1b3a2e"). InnerVolumeSpecName "kube-api-access-b6k9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238058    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238101    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238115    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238129    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238142    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238170    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238182    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238195    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6k9m\" (UniqueName: \"kubernetes.io/projected/652750b3-8ca9-481a-b675-1ee10b1b3a2e-kube-api-access-b6k9m\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238209    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238223    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9spz\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-kube-api-access-k9spz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238237    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238251    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2tr\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-kube-api-access-nb2tr\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238263    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238278    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238290    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238303    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238316    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238328    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238343    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238355    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238368    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432e6150-2d0b-4e0a-9514-08e0fec591d7-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238381    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238394    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238407    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238419    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238432    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dw27\" (UniqueName: \"kubernetes.io/projected/7959c1d6-8d4c-478f-86e6-e4795aa0f110-kube-api-access-5dw27\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238448    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238462    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432e6150-2d0b-4e0a-9514-08e0fec591d7-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238475    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238487    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652750b3-8ca9-481a-b675-1ee10b1b3a2e-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.239029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.238498    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.240520 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.240486    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432e6150-2d0b-4e0a-9514-08e0fec591d7-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.240596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.240528    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.240596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.240542    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.240596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.240557    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59caa8f3-3d24-4f7a-8d4f-78914d2cef09-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.240596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.240571    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7959c1d6-8d4c-478f-86e6-e4795aa0f110-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:32:47.730540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.730506    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"
Oct 30 21:32:47.731019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.730539    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc" event={"ID":"652750b3-8ca9-481a-b675-1ee10b1b3a2e","Type":"ContainerDied","Data":"09d49237ecb50b30d21484e1656c8fcb169c43f3517e0f1ba4cb32893283374b"}
Oct 30 21:32:47.731019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.730624    5886 scope.go:122] "RemoveContainer" containerID="eea3e934e803696f2b9c252c28eee4c3d1e989b13773aff45624a926138f71d4"
Oct 30 21:32:47.732270 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.732231    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc" event={"ID":"59caa8f3-3d24-4f7a-8d4f-78914d2cef09","Type":"ContainerDied","Data":"a655f3e8bbfe9000994b53edd1ef7c6e7dfb07213c3494d9edcaa51ed88d61c8"}
Oct 30 21:32:47.732404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.732219    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"
Oct 30 21:32:47.733803 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.733782    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"
Oct 30 21:32:47.733924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.733801    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp" event={"ID":"7959c1d6-8d4c-478f-86e6-e4795aa0f110","Type":"ContainerDied","Data":"bda6864b2f9687ba5be3c21d89a084b2753ec749ad4e99eda159b9af386b3115"}
Oct 30 21:32:47.735461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.735435    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk" event={"ID":"432e6150-2d0b-4e0a-9514-08e0fec591d7","Type":"ContainerDied","Data":"018f32d8ed12f06d74fbe893a8f50a2f4a3c99262c49481545e0eb1f585859b7"}
Oct 30 21:32:47.735461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.735451    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"
Oct 30 21:32:47.744017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.743995    5886 scope.go:122] "RemoveContainer" containerID="351ff491b0b6149e0634c16e59a92aa4545ed1d52701b52105d19de896018759"
Oct 30 21:32:47.755596 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.755577    5886 scope.go:122] "RemoveContainer" containerID="c30f61be4b801a21c83e943cbd8b0a4752db3f6582aeac77720ccb74933c0d96"
Oct 30 21:32:47.764884 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.764858    5886 scope.go:122] "RemoveContainer" containerID="f600f84467eacb671c4e93d525adb96b913947a1f0a4bb9f72bc65b907546117"
Oct 30 21:32:47.810670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.810641    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"]
Oct 30 21:32:47.821702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.821675    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-4hjmc"]
Oct 30 21:32:47.867069 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.867045    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"]
Oct 30 21:32:47.870538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.870515    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-txthk"]
Oct 30 21:32:47.901828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.901797    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"]
Oct 30 21:32:47.907961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.907937    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-g8lwc"]
Oct 30 21:32:47.938970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.938941    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"]
Oct 30 21:32:47.942470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.942437    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-8ltrp"]
Oct 30 21:32:47.989934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.989905    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" path="/var/lib/kubelet/pods/432e6150-2d0b-4e0a-9514-08e0fec591d7/volumes"
Oct 30 21:32:47.990429 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.990414    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" path="/var/lib/kubelet/pods/59caa8f3-3d24-4f7a-8d4f-78914d2cef09/volumes"
Oct 30 21:32:47.990918 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.990897    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" path="/var/lib/kubelet/pods/652750b3-8ca9-481a-b675-1ee10b1b3a2e/volumes"
Oct 30 21:32:47.991696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:47.991680    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" path="/var/lib/kubelet/pods/7959c1d6-8d4c-478f-86e6-e4795aa0f110/volumes"
Oct 30 21:32:49.147399 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.147362    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"]
Oct 30 21:32:49.148007 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.147993    5886 cpu_manager.go:414] "containerMap: removing container" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy"
Oct 30 21:32:49.148054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148007    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy"
Oct 30 21:32:49.148054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148033    5886 cpu_manager.go:414] "containerMap: removing container" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy"
Oct 30 21:32:49.148054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148038    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy"
Oct 30 21:32:49.148054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148047    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy"
Oct 30 21:32:49.148054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148053    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy"
Oct 30 21:32:49.148246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148066    5886 cpu_manager.go:414] "containerMap: removing container" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy"
Oct 30 21:32:49.148246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148072    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy"
Oct 30 21:32:49.148246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148181    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="432e6150-2d0b-4e0a-9514-08e0fec591d7" containerName="istio-proxy"
Oct 30 21:32:49.148246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148193    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="59caa8f3-3d24-4f7a-8d4f-78914d2cef09" containerName="istio-proxy"
Oct 30 21:32:49.148246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148199    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7959c1d6-8d4c-478f-86e6-e4795aa0f110" containerName="istio-proxy"
Oct 30 21:32:49.148246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.148206    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="652750b3-8ca9-481a-b675-1ee10b1b3a2e" containerName="istio-proxy"
Oct 30 21:32:49.151913 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.151889    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.168041 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.168012    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e38f60-serving-cert\""
Oct 30 21:32:49.186392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.185526    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"]
Oct 30 21:32:49.268671 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.268636    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8cl\" (UniqueName: \"kubernetes.io/projected/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-kube-api-access-qr8cl\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.268881 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.268807    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.269007 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.268986    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.269085 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.269068    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-metrics-certs\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.370046 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.370001    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.370281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.370150    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.370281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.370260    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-metrics-certs\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.370412 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.370346    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8cl\" (UniqueName: \"kubernetes.io/projected/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-kube-api-access-qr8cl\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.370781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.370756    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.372632 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.372608    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-metrics-certs\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.372801 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.372664    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.388912 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.388885    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8cl\" (UniqueName: \"kubernetes.io/projected/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-kube-api-access-qr8cl\") pod \"maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.466640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.466551    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:49.779822 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.779747    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"]
Oct 30 21:32:49.782570 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:32:49.782540    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f6e9d2_1bb2_4b2c_bbee_95bc89183d5f.slice/crio-4e4162d91026105f1f36f1feb4a765e19ac7528cad848b1c2063f9793299b24f WatchSource:0}: Error finding container 4e4162d91026105f1f36f1feb4a765e19ac7528cad848b1c2063f9793299b24f: Status 404 returned error can't find the container with id 4e4162d91026105f1f36f1feb4a765e19ac7528cad848b1c2063f9793299b24f
Oct 30 21:32:49.784423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:49.784400    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:32:50.775407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:50.775354    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l" event={"ID":"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f","Type":"ContainerStarted","Data":"4737076ba518caf69b3f5f6052859be1efa503b905a2d9661d2cd55634ccf9a0"}
Oct 30 21:32:50.775789 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:50.775430    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:50.775789 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:50.775453    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l" event={"ID":"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f","Type":"ContainerStarted","Data":"4e4162d91026105f1f36f1feb4a765e19ac7528cad848b1c2063f9793299b24f"}
Oct 30 21:32:50.815686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:50.815626    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l" podStartSLOduration=1.431085287 podStartE2EDuration="1.815609974s" totalImagesPullingTime="384.524687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:50.814414568 +0000 UTC m=+1370.943666026" watchObservedRunningTime="2026-10-30 21:32:50.815609974 +0000 UTC m=+1370.944861434"
Oct 30 21:32:54.818913 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:54.818879    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"]
Oct 30 21:32:54.823537 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:54.823517    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:54.849118 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:54.849087    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b1659a-serving-cert\""
Oct 30 21:32:54.864591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:54.854249    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"]
Oct 30 21:32:54.942932 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:54.942892    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6vv\" (UniqueName: \"kubernetes.io/projected/517c9e29-f434-496f-ad7d-8650e88be247-kube-api-access-qc6vv\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:54.943124 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:54.943099    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-maas-api-tls\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:54.943262 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:54.943244    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c9e29-f434-496f-ad7d-8650e88be247-openshift-service-ca\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:54.943330 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:54.943311    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-metrics-certs\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.051484 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.051448    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6vv\" (UniqueName: \"kubernetes.io/projected/517c9e29-f434-496f-ad7d-8650e88be247-kube-api-access-qc6vv\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.051704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.051679    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-maas-api-tls\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.051820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.051801    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c9e29-f434-496f-ad7d-8650e88be247-openshift-service-ca\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.051887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.051859    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-metrics-certs\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.053473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.053442    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c9e29-f434-496f-ad7d-8650e88be247-openshift-service-ca\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.055533 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.055505    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-metrics-certs\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.055746 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.055532    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-maas-api-tls\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.071639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.071565    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6vv\" (UniqueName: \"kubernetes.io/projected/517c9e29-f434-496f-ad7d-8650e88be247-kube-api-access-qc6vv\") pod \"maas-api-e2e-api-b-b1659a-6974fb4486-mthk6\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.138124 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.138072    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:55.452652 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.452621    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"]
Oct 30 21:32:55.456649 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:32:55.456617    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517c9e29_f434_496f_ad7d_8650e88be247.slice/crio-4e2f656c9b1644c7aa8c82b13f0a8dac6ec8e728d65997dc340201e0c85a889e WatchSource:0}: Error finding container 4e2f656c9b1644c7aa8c82b13f0a8dac6ec8e728d65997dc340201e0c85a889e: Status 404 returned error can't find the container with id 4e2f656c9b1644c7aa8c82b13f0a8dac6ec8e728d65997dc340201e0c85a889e
Oct 30 21:32:55.818067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:55.818037    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6" event={"ID":"517c9e29-f434-496f-ad7d-8650e88be247","Type":"ContainerStarted","Data":"4e2f656c9b1644c7aa8c82b13f0a8dac6ec8e728d65997dc340201e0c85a889e"}
Oct 30 21:32:56.801181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:56.791619    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:32:56.843195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:56.834656    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6" event={"ID":"517c9e29-f434-496f-ad7d-8650e88be247","Type":"ContainerStarted","Data":"c187eb212c063c446d54b48f721b0c3e5574b8ea28949401b0ac79bdfd8ad2c1"}
Oct 30 21:32:56.843195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:56.834751    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:32:56.905902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:56.905838    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6" podStartSLOduration=2.569353487 podStartE2EDuration="2.905817076s" totalImagesPullingTime="336.463589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:56.899575917 +0000 UTC m=+1377.028827376" watchObservedRunningTime="2026-10-30 21:32:56.905817076 +0000 UTC m=+1377.035068534"
Oct 30 21:32:58.881940 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:32:58.881903    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"]
Oct 30 21:33:02.847456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:02.847426    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:33:03.882609 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:03.882568    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf" podUID="b25778ff-68d1-41dc-ba31-d64258679d40" containerName="payload-pre-processing" containerID="cri-o://fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886" gracePeriod=25
Oct 30 21:33:04.164091 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.164055    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:33:04.287277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.287242    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25778ff-68d1-41dc-ba31-d64258679d40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25778ff-68d1-41dc-ba31-d64258679d40-plugins-config-volume\") pod \"b25778ff-68d1-41dc-ba31-d64258679d40\" (UID: \"b25778ff-68d1-41dc-ba31-d64258679d40\") "
Oct 30 21:33:04.287547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.287523    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25778ff-68d1-41dc-ba31-d64258679d40-kube-api-access-kwgmp\" (UniqueName: \"kubernetes.io/projected/b25778ff-68d1-41dc-ba31-d64258679d40-kube-api-access-kwgmp\") pod \"b25778ff-68d1-41dc-ba31-d64258679d40\" (UID: \"b25778ff-68d1-41dc-ba31-d64258679d40\") "
Oct 30 21:33:04.287722 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.287697    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25778ff-68d1-41dc-ba31-d64258679d40-plugins-config-volume" pod "b25778ff-68d1-41dc-ba31-d64258679d40" (UID: "b25778ff-68d1-41dc-ba31-d64258679d40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:04.288108 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.288088    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25778ff-68d1-41dc-ba31-d64258679d40-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:04.289850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.289822    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25778ff-68d1-41dc-ba31-d64258679d40-kube-api-access-kwgmp" pod "b25778ff-68d1-41dc-ba31-d64258679d40" (UID: "b25778ff-68d1-41dc-ba31-d64258679d40"). InnerVolumeSpecName "kube-api-access-kwgmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:04.389021 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.388948    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwgmp\" (UniqueName: \"kubernetes.io/projected/b25778ff-68d1-41dc-ba31-d64258679d40-kube-api-access-kwgmp\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:04.898875 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.898835    5886 generic.go:356] "Generic (PLEG): container finished" podID="b25778ff-68d1-41dc-ba31-d64258679d40" containerID="fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886" exitCode=0
Oct 30 21:33:04.899392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.898887    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"
Oct 30 21:33:04.899392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.898904    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf" event={"ID":"b25778ff-68d1-41dc-ba31-d64258679d40","Type":"ContainerDied","Data":"fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886"}
Oct 30 21:33:04.899392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.898974    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf" event={"ID":"b25778ff-68d1-41dc-ba31-d64258679d40","Type":"ContainerDied","Data":"654feb566de1fe5d44f12971f65b89ab33c48c898119e2f2a3e35d728e3661a6"}
Oct 30 21:33:04.899392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.899004    5886 scope.go:122] "RemoveContainer" containerID="fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886"
Oct 30 21:33:04.914313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.914285    5886 scope.go:122] "RemoveContainer" containerID="fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886"
Oct 30 21:33:04.914635 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:33:04.914615    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886\": container with ID starting with fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886 not found: ID does not exist" containerID="fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886"
Oct 30 21:33:04.914707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.914644    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886"} err="failed to get container status \"fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886\": rpc error: code = NotFound desc = could not find container \"fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886\": container with ID starting with fbd50df8910076b3508711e37e8acf7a891c87853f71fc2fd5425ef88a2f1886 not found: ID does not exist"
Oct 30 21:33:04.948313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.948285    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"]
Oct 30 21:33:04.959708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:04.959679    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a1ee3-5596cb64dc-84bjf"]
Oct 30 21:33:05.990702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:05.990658    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25778ff-68d1-41dc-ba31-d64258679d40" path="/var/lib/kubelet/pods/b25778ff-68d1-41dc-ba31-d64258679d40/volumes"
Oct 30 21:33:09.191218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:09.191177    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"]
Oct 30 21:33:11.029896 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:33:11.029854    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:33:14.191852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.191811    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb" podUID="22fdc49f-ebc3-497a-bb3c-39e71ca3da5a" containerName="payload-processing" containerID="cri-o://6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d" gracePeriod=25
Oct 30 21:33:14.467777 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.467751    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:33:14.542508 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.542469    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-plugins-config-volume\") pod \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\" (UID: \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\") "
Oct 30 21:33:14.542808 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.542559    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-kube-api-access-z4rkq\" (UniqueName: \"kubernetes.io/projected/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-kube-api-access-z4rkq\") pod \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\" (UID: \"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a\") "
Oct 30 21:33:14.543420 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.543377    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-plugins-config-volume" pod "22fdc49f-ebc3-497a-bb3c-39e71ca3da5a" (UID: "22fdc49f-ebc3-497a-bb3c-39e71ca3da5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:14.543547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.543525    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:14.548427 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.548395    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-kube-api-access-z4rkq" pod "22fdc49f-ebc3-497a-bb3c-39e71ca3da5a" (UID: "22fdc49f-ebc3-497a-bb3c-39e71ca3da5a"). InnerVolumeSpecName "kube-api-access-z4rkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:14.645086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.645049    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4rkq\" (UniqueName: \"kubernetes.io/projected/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a-kube-api-access-z4rkq\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:14.989037 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.989005    5886 generic.go:356] "Generic (PLEG): container finished" podID="22fdc49f-ebc3-497a-bb3c-39e71ca3da5a" containerID="6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d" exitCode=0
Oct 30 21:33:14.989258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.989067    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"
Oct 30 21:33:14.989258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.989089    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb" event={"ID":"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a","Type":"ContainerDied","Data":"6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d"}
Oct 30 21:33:14.989258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.989147    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb" event={"ID":"22fdc49f-ebc3-497a-bb3c-39e71ca3da5a","Type":"ContainerDied","Data":"b0b5ae1ea1b7683170fe00ceaedc6cf661e8abffba79ad4672fdc7d8df9dd21a"}
Oct 30 21:33:14.989258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.989171    5886 scope.go:122] "RemoveContainer" containerID="6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d"
Oct 30 21:33:15.000005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:14.999989    5886 scope.go:122] "RemoveContainer" containerID="6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d"
Oct 30 21:33:15.000293 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:33:15.000272    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d\": container with ID starting with 6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d not found: ID does not exist" containerID="6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d"
Oct 30 21:33:15.000344 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:15.000300    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d"} err="failed to get container status \"6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d\": rpc error: code = NotFound desc = could not find container \"6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d\": container with ID starting with 6e3ffe5c68d9434d6234e3a20e55d41050584d432a436d98d35342ed6e81808d not found: ID does not exist"
Oct 30 21:33:15.039048 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:15.039009    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"]
Oct 30 21:33:15.042939 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:15.042909    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-6bd7885cb7-z55lb"]
Oct 30 21:33:15.990146 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:15.990114    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fdc49f-ebc3-497a-bb3c-39e71ca3da5a" path="/var/lib/kubelet/pods/22fdc49f-ebc3-497a-bb3c-39e71ca3da5a/volumes"
Oct 30 21:33:23.517393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:23.517349    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"]
Oct 30 21:33:23.555178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:23.552034    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"]
Oct 30 21:33:28.517816 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:28.517761    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc" podUID="ff30c120-098b-45d9-8dd5-0cc84e53e308" containerName="payload-processing" containerID="cri-o://7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e" gracePeriod=25
Oct 30 21:33:28.553207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:28.553104    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc" podUID="dd13dc0a-2bc5-4d7b-9d18-837f1e301efc" containerName="payload-pre-processing" containerID="cri-o://8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027" gracePeriod=25
Oct 30 21:33:28.862092 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:28.862062    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:33:28.935125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:28.935097    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:33:28.954527 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:28.954501    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff30c120-098b-45d9-8dd5-0cc84e53e308-kube-api-access-rx462\" (UniqueName: \"kubernetes.io/projected/ff30c120-098b-45d9-8dd5-0cc84e53e308-kube-api-access-rx462\") pod \"ff30c120-098b-45d9-8dd5-0cc84e53e308\" (UID: \"ff30c120-098b-45d9-8dd5-0cc84e53e308\") "
Oct 30 21:33:28.954693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:28.954624    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff30c120-098b-45d9-8dd5-0cc84e53e308-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff30c120-098b-45d9-8dd5-0cc84e53e308-plugins-config-volume\") pod \"ff30c120-098b-45d9-8dd5-0cc84e53e308\" (UID: \"ff30c120-098b-45d9-8dd5-0cc84e53e308\") "
Oct 30 21:33:28.955045 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:28.955013    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff30c120-098b-45d9-8dd5-0cc84e53e308-plugins-config-volume" pod "ff30c120-098b-45d9-8dd5-0cc84e53e308" (UID: "ff30c120-098b-45d9-8dd5-0cc84e53e308"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:28.956835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:28.956802    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff30c120-098b-45d9-8dd5-0cc84e53e308-kube-api-access-rx462" pod "ff30c120-098b-45d9-8dd5-0cc84e53e308" (UID: "ff30c120-098b-45d9-8dd5-0cc84e53e308"). InnerVolumeSpecName "kube-api-access-rx462". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:29.056148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.056118    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-plugins-config-volume\") pod \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\" (UID: \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\") "
Oct 30 21:33:29.056355 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.056324    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-kube-api-access-w6rns\" (UniqueName: \"kubernetes.io/projected/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-kube-api-access-w6rns\") pod \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\" (UID: \"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc\") "
Oct 30 21:33:29.056501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.056476    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-plugins-config-volume" pod "dd13dc0a-2bc5-4d7b-9d18-837f1e301efc" (UID: "dd13dc0a-2bc5-4d7b-9d18-837f1e301efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:29.057331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.057302    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff30c120-098b-45d9-8dd5-0cc84e53e308-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:29.057331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.057331    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:29.057513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.057347    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx462\" (UniqueName: \"kubernetes.io/projected/ff30c120-098b-45d9-8dd5-0cc84e53e308-kube-api-access-rx462\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:29.058511 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.058480    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-kube-api-access-w6rns" pod "dd13dc0a-2bc5-4d7b-9d18-837f1e301efc" (UID: "dd13dc0a-2bc5-4d7b-9d18-837f1e301efc"). InnerVolumeSpecName "kube-api-access-w6rns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:29.107237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.107140    5886 generic.go:356] "Generic (PLEG): container finished" podID="ff30c120-098b-45d9-8dd5-0cc84e53e308" containerID="7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e" exitCode=0
Oct 30 21:33:29.107237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.107201    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"
Oct 30 21:33:29.107463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.107231    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc" event={"ID":"ff30c120-098b-45d9-8dd5-0cc84e53e308","Type":"ContainerDied","Data":"7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e"}
Oct 30 21:33:29.107463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.107292    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc" event={"ID":"ff30c120-098b-45d9-8dd5-0cc84e53e308","Type":"ContainerDied","Data":"ac3640df9ec6154c55ad18a777f6a12a78803293f85ddc677f34d3699144091c"}
Oct 30 21:33:29.107463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.107321    5886 scope.go:122] "RemoveContainer" containerID="7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e"
Oct 30 21:33:29.108961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.108940    5886 generic.go:356] "Generic (PLEG): container finished" podID="dd13dc0a-2bc5-4d7b-9d18-837f1e301efc" containerID="8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027" exitCode=0
Oct 30 21:33:29.109087 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.108986    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"
Oct 30 21:33:29.109087 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.109010    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc" event={"ID":"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc","Type":"ContainerDied","Data":"8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027"}
Oct 30 21:33:29.109087 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.109055    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc" event={"ID":"dd13dc0a-2bc5-4d7b-9d18-837f1e301efc","Type":"ContainerDied","Data":"ca5df42c59dd972e238fe9d1c5e0d6076efa78bbddd85d44c540f042378d4f5a"}
Oct 30 21:33:29.121663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.121646    5886 scope.go:122] "RemoveContainer" containerID="7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e"
Oct 30 21:33:29.121945 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:33:29.121925    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e\": container with ID starting with 7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e not found: ID does not exist" containerID="7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e"
Oct 30 21:33:29.122000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.121954    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e"} err="failed to get container status \"7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e\": rpc error: code = NotFound desc = could not find container \"7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e\": container with ID starting with 7fcfcf5476a9c1df3394cec17ec7d9cec384a8a691ee54e2c4c2d697f071232e not found: ID does not exist"
Oct 30 21:33:29.122000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.121972    5886 scope.go:122] "RemoveContainer" containerID="8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027"
Oct 30 21:33:29.131712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.131694    5886 scope.go:122] "RemoveContainer" containerID="8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027"
Oct 30 21:33:29.131958 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:33:29.131938    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027\": container with ID starting with 8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027 not found: ID does not exist" containerID="8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027"
Oct 30 21:33:29.132005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.131967    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027"} err="failed to get container status \"8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027\": rpc error: code = NotFound desc = could not find container \"8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027\": container with ID starting with 8f809ca785f94f93f95eb795a98189cdc91b714199c4323a332bede02cb46027 not found: ID does not exist"
Oct 30 21:33:29.156438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.156410    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"]
Oct 30 21:33:29.158421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.158399    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rns\" (UniqueName: \"kubernetes.io/projected/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc-kube-api-access-w6rns\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:29.166072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.166043    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-59875d74cf-srmvc"]
Oct 30 21:33:29.203393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.200047    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"]
Oct 30 21:33:29.205144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.205112    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-666d7b9b79-7w2cc"]
Oct 30 21:33:29.990546 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.990512    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd13dc0a-2bc5-4d7b-9d18-837f1e301efc" path="/var/lib/kubelet/pods/dd13dc0a-2bc5-4d7b-9d18-837f1e301efc/volumes"
Oct 30 21:33:29.991082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:29.991066    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff30c120-098b-45d9-8dd5-0cc84e53e308" path="/var/lib/kubelet/pods/ff30c120-098b-45d9-8dd5-0cc84e53e308/volumes"
Oct 30 21:33:36.227358 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.227324    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"]
Oct 30 21:33:36.227966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.227949    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b25778ff-68d1-41dc-ba31-d64258679d40" containerName="payload-pre-processing"
Oct 30 21:33:36.228009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.227967    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25778ff-68d1-41dc-ba31-d64258679d40" containerName="payload-pre-processing"
Oct 30 21:33:36.228009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.227993    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ff30c120-098b-45d9-8dd5-0cc84e53e308" containerName="payload-processing"
Oct 30 21:33:36.228009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.227999    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff30c120-098b-45d9-8dd5-0cc84e53e308" containerName="payload-processing"
Oct 30 21:33:36.228009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.228007    5886 cpu_manager.go:414] "containerMap: removing container" podUID="dd13dc0a-2bc5-4d7b-9d18-837f1e301efc" containerName="payload-pre-processing"
Oct 30 21:33:36.228145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.228013    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd13dc0a-2bc5-4d7b-9d18-837f1e301efc" containerName="payload-pre-processing"
Oct 30 21:33:36.228145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.228039    5886 cpu_manager.go:414] "containerMap: removing container" podUID="22fdc49f-ebc3-497a-bb3c-39e71ca3da5a" containerName="payload-processing"
Oct 30 21:33:36.228145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.228045    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fdc49f-ebc3-497a-bb3c-39e71ca3da5a" containerName="payload-processing"
Oct 30 21:33:36.228145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.228135    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff30c120-098b-45d9-8dd5-0cc84e53e308" containerName="payload-processing"
Oct 30 21:33:36.228145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.228144    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd13dc0a-2bc5-4d7b-9d18-837f1e301efc" containerName="payload-pre-processing"
Oct 30 21:33:36.228316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.228169    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fdc49f-ebc3-497a-bb3c-39e71ca3da5a" containerName="payload-processing"
Oct 30 21:33:36.228316 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.228179    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25778ff-68d1-41dc-ba31-d64258679d40" containerName="payload-pre-processing"
Oct 30 21:33:36.231863 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.231844    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:36.248367 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.248345    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7222e2\""
Oct 30 21:33:36.265722 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.265673    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"]
Oct 30 21:33:36.301261 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.301230    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7222e2-dockercfg-82pqw\""
Oct 30 21:33:36.364580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.364547    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9tdm\" (UniqueName: \"kubernetes.io/projected/6628b759-f9c9-4d04-a472-2bbf88be896b-kube-api-access-j9tdm\") pod \"payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp\" (UID: \"6628b759-f9c9-4d04-a472-2bbf88be896b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:36.364829 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.364802    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6628b759-f9c9-4d04-a472-2bbf88be896b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp\" (UID: \"6628b759-f9c9-4d04-a472-2bbf88be896b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:36.466091 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.466048    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6628b759-f9c9-4d04-a472-2bbf88be896b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp\" (UID: \"6628b759-f9c9-4d04-a472-2bbf88be896b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:36.466315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.466203    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9tdm\" (UniqueName: \"kubernetes.io/projected/6628b759-f9c9-4d04-a472-2bbf88be896b-kube-api-access-j9tdm\") pod \"payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp\" (UID: \"6628b759-f9c9-4d04-a472-2bbf88be896b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:36.467019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.466992    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6628b759-f9c9-4d04-a472-2bbf88be896b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp\" (UID: \"6628b759-f9c9-4d04-a472-2bbf88be896b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:36.484895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.484830    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9tdm\" (UniqueName: \"kubernetes.io/projected/6628b759-f9c9-4d04-a472-2bbf88be896b-kube-api-access-j9tdm\") pod \"payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp\" (UID: \"6628b759-f9c9-4d04-a472-2bbf88be896b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:36.545381 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.545352    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:36.876764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:36.875303    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"]
Oct 30 21:33:36.876764 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:33:36.875499    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6628b759_f9c9_4d04_a472_2bbf88be896b.slice/crio-08e6b503ec6c470ea9d33b73d5a6ec40cdc5cd7dc76d07dfabefb55deb67507f WatchSource:0}: Error finding container 08e6b503ec6c470ea9d33b73d5a6ec40cdc5cd7dc76d07dfabefb55deb67507f: Status 404 returned error can't find the container with id 08e6b503ec6c470ea9d33b73d5a6ec40cdc5cd7dc76d07dfabefb55deb67507f
Oct 30 21:33:37.178087 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:37.178040    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp" event={"ID":"6628b759-f9c9-4d04-a472-2bbf88be896b","Type":"ContainerStarted","Data":"2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec"}
Oct 30 21:33:37.178310 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:37.178104    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp" event={"ID":"6628b759-f9c9-4d04-a472-2bbf88be896b","Type":"ContainerStarted","Data":"08e6b503ec6c470ea9d33b73d5a6ec40cdc5cd7dc76d07dfabefb55deb67507f"}
Oct 30 21:33:37.179059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:37.179033    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:37.216528 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:37.216480    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp" podStartSLOduration=1.216465787 podStartE2EDuration="1.216465787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:37.215737802 +0000 UTC m=+1417.344989262" watchObservedRunningTime="2026-10-30 21:33:37.216465787 +0000 UTC m=+1417.345717245"
Oct 30 21:33:38.637742 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:38.637705    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"]
Oct 30 21:33:38.673974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:38.673934    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"]
Oct 30 21:33:41.032397 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:33:41.032362    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:33:42.855274 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:42.855239    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"]
Oct 30 21:33:42.860043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:42.860021    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:42.880723 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:42.880679    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"]
Oct 30 21:33:42.898570 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:42.898544    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b0e991\""
Oct 30 21:33:42.898570 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:42.898546    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b0e991-dockercfg-845x7\""
Oct 30 21:33:42.969182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:42.967252    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc\" (UID: \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:42.969182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:42.967518    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cknhf\" (UniqueName: \"kubernetes.io/projected/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-kube-api-access-cknhf\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc\" (UID: \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:43.068501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.068464    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc\" (UID: \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:43.068927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.068894    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cknhf\" (UniqueName: \"kubernetes.io/projected/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-kube-api-access-cknhf\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc\" (UID: \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:43.072370 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.070502    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc\" (UID: \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:43.087641 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.087599    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cknhf\" (UniqueName: \"kubernetes.io/projected/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-kube-api-access-cknhf\") pod \"payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc\" (UID: \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:43.175325 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.175292    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:43.491056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.490904    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"]
Oct 30 21:33:43.494058 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:33:43.494029    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f28f61_0c8b_411a_8aaa_f4a2b84511eb.slice/crio-b5b9952955ddb9c9c2d958637f69bd95d24a19400d5f0da86cab5a22c78a0207 WatchSource:0}: Error finding container b5b9952955ddb9c9c2d958637f69bd95d24a19400d5f0da86cab5a22c78a0207: Status 404 returned error can't find the container with id b5b9952955ddb9c9c2d958637f69bd95d24a19400d5f0da86cab5a22c78a0207
Oct 30 21:33:43.638185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.638123    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx" podUID="953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c" containerName="payload-processing" containerID="cri-o://94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6" gracePeriod=25
Oct 30 21:33:43.674424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.674389    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5" podUID="3b0a790f-42eb-4a41-aae9-718a87ef805f" containerName="payload-pre-processing" containerID="cri-o://01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c" gracePeriod=25
Oct 30 21:33:43.957408 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:43.957382    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:33:44.053833 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.053769    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:33:44.082235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.082199    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-kube-api-access-jr2hc\" (UniqueName: \"kubernetes.io/projected/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-kube-api-access-jr2hc\") pod \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\" (UID: \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\") "
Oct 30 21:33:44.082385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.082292    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-plugins-config-volume\") pod \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\" (UID: \"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c\") "
Oct 30 21:33:44.083055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.083027    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-plugins-config-volume" pod "953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c" (UID: "953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:44.084942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.084918    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-kube-api-access-jr2hc" pod "953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c" (UID: "953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c"). InnerVolumeSpecName "kube-api-access-jr2hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:44.183786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.183752    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0a790f-42eb-4a41-aae9-718a87ef805f-kube-api-access-fkknt\" (UniqueName: \"kubernetes.io/projected/3b0a790f-42eb-4a41-aae9-718a87ef805f-kube-api-access-fkknt\") pod \"3b0a790f-42eb-4a41-aae9-718a87ef805f\" (UID: \"3b0a790f-42eb-4a41-aae9-718a87ef805f\") "
Oct 30 21:33:44.183981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.183817    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0a790f-42eb-4a41-aae9-718a87ef805f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0a790f-42eb-4a41-aae9-718a87ef805f-plugins-config-volume\") pod \"3b0a790f-42eb-4a41-aae9-718a87ef805f\" (UID: \"3b0a790f-42eb-4a41-aae9-718a87ef805f\") "
Oct 30 21:33:44.184252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.184218    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0a790f-42eb-4a41-aae9-718a87ef805f-plugins-config-volume" pod "3b0a790f-42eb-4a41-aae9-718a87ef805f" (UID: "3b0a790f-42eb-4a41-aae9-718a87ef805f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:44.184365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.184312    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0a790f-42eb-4a41-aae9-718a87ef805f-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:44.184365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.184329    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr2hc\" (UniqueName: \"kubernetes.io/projected/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-kube-api-access-jr2hc\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:44.184365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.184339    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:44.186001 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.185971    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0a790f-42eb-4a41-aae9-718a87ef805f-kube-api-access-fkknt" pod "3b0a790f-42eb-4a41-aae9-718a87ef805f" (UID: "3b0a790f-42eb-4a41-aae9-718a87ef805f"). InnerVolumeSpecName "kube-api-access-fkknt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:44.234570 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.234531    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc" event={"ID":"02f28f61-0c8b-411a-8aaa-f4a2b84511eb","Type":"ContainerStarted","Data":"62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52"}
Oct 30 21:33:44.234769 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.234616    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc" event={"ID":"02f28f61-0c8b-411a-8aaa-f4a2b84511eb","Type":"ContainerStarted","Data":"b5b9952955ddb9c9c2d958637f69bd95d24a19400d5f0da86cab5a22c78a0207"}
Oct 30 21:33:44.234769 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.234642    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:44.235955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.235920    5886 generic.go:356] "Generic (PLEG): container finished" podID="953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c" containerID="94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6" exitCode=0
Oct 30 21:33:44.236059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.235993    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"
Oct 30 21:33:44.236059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.236014    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx" event={"ID":"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c","Type":"ContainerDied","Data":"94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6"}
Oct 30 21:33:44.236170 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.236062    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx" event={"ID":"953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c","Type":"ContainerDied","Data":"02bdf76bc8788dd6b291036fa11956c4e531e68dc65f862bbe52b0def8d2bbea"}
Oct 30 21:33:44.236170 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.236094    5886 scope.go:122] "RemoveContainer" containerID="94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6"
Oct 30 21:33:44.237787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.237740    5886 generic.go:356] "Generic (PLEG): container finished" podID="3b0a790f-42eb-4a41-aae9-718a87ef805f" containerID="01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c" exitCode=0
Oct 30 21:33:44.237907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.237800    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5" event={"ID":"3b0a790f-42eb-4a41-aae9-718a87ef805f","Type":"ContainerDied","Data":"01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c"}
Oct 30 21:33:44.237907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.237835    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5" event={"ID":"3b0a790f-42eb-4a41-aae9-718a87ef805f","Type":"ContainerDied","Data":"db18332e25ea8e3bfe2c33e4f28ffa26b227c6469039a70f788d103713904eef"}
Oct 30 21:33:44.237907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.237802    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"
Oct 30 21:33:44.247200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.247182    5886 scope.go:122] "RemoveContainer" containerID="94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6"
Oct 30 21:33:44.247511 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:33:44.247484    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6\": container with ID starting with 94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6 not found: ID does not exist" containerID="94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6"
Oct 30 21:33:44.247588 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.247518    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6"} err="failed to get container status \"94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6\": rpc error: code = NotFound desc = could not find container \"94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6\": container with ID starting with 94b5ada78bc376e304c41b8b9f55b847cbb87dc3fc9c56002de6f79f11fbaad6 not found: ID does not exist"
Oct 30 21:33:44.247588 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.247538    5886 scope.go:122] "RemoveContainer" containerID="01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c"
Oct 30 21:33:44.256450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.256432    5886 scope.go:122] "RemoveContainer" containerID="01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c"
Oct 30 21:33:44.256694 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:33:44.256676    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c\": container with ID starting with 01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c not found: ID does not exist" containerID="01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c"
Oct 30 21:33:44.256760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.256701    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c"} err="failed to get container status \"01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c\": rpc error: code = NotFound desc = could not find container \"01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c\": container with ID starting with 01988aa96dc4112093b18031976e3f1f63e5ed6c01ef4d04cfebbf640aa78f9c not found: ID does not exist"
Oct 30 21:33:44.272725 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.272684    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc" podStartSLOduration=2.272672404 podStartE2EDuration="2.272672404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:44.269218684 +0000 UTC m=+1424.398470133" watchObservedRunningTime="2026-10-30 21:33:44.272672404 +0000 UTC m=+1424.401923861"
Oct 30 21:33:44.285374 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.285349    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkknt\" (UniqueName: \"kubernetes.io/projected/3b0a790f-42eb-4a41-aae9-718a87ef805f-kube-api-access-fkknt\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:33:44.306215 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.306128    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"]
Oct 30 21:33:44.309886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.309854    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-18a043-6ccdf4cdc-vr5f5"]
Oct 30 21:33:44.341328 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.341301    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"]
Oct 30 21:33:44.345227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:44.345201    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-18a043-7575bd895c-prbxx"]
Oct 30 21:33:45.625852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.625814    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"]
Oct 30 21:33:45.626807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.626784    5886 cpu_manager.go:414] "containerMap: removing container" podUID="953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c" containerName="payload-processing"
Oct 30 21:33:45.626807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.626808    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c" containerName="payload-processing"
Oct 30 21:33:45.626980 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.626836    5886 cpu_manager.go:414] "containerMap: removing container" podUID="3b0a790f-42eb-4a41-aae9-718a87ef805f" containerName="payload-pre-processing"
Oct 30 21:33:45.626980 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.626843    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0a790f-42eb-4a41-aae9-718a87ef805f" containerName="payload-pre-processing"
Oct 30 21:33:45.626980 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.626988    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c" containerName="payload-processing"
Oct 30 21:33:45.627135 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.627001    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0a790f-42eb-4a41-aae9-718a87ef805f" containerName="payload-pre-processing"
Oct 30 21:33:45.658212 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.655895    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:45.658212 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.656096    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"]
Oct 30 21:33:45.670860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.670828    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e38f60\""
Oct 30 21:33:45.713200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.712788    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xbx\" (UniqueName: \"kubernetes.io/projected/621bb7f9-3d3b-47df-8a4f-95109761ecac-kube-api-access-29xbx\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d\" (UID: \"621bb7f9-3d3b-47df-8a4f-95109761ecac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:45.713200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.712865    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621bb7f9-3d3b-47df-8a4f-95109761ecac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d\" (UID: \"621bb7f9-3d3b-47df-8a4f-95109761ecac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:45.801791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.801760    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"]
Oct 30 21:33:45.806314 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.806288    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:45.814830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.814798    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xbx\" (UniqueName: \"kubernetes.io/projected/621bb7f9-3d3b-47df-8a4f-95109761ecac-kube-api-access-29xbx\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d\" (UID: \"621bb7f9-3d3b-47df-8a4f-95109761ecac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:45.814964 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.814859    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621bb7f9-3d3b-47df-8a4f-95109761ecac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d\" (UID: \"621bb7f9-3d3b-47df-8a4f-95109761ecac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:45.815835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.815814    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621bb7f9-3d3b-47df-8a4f-95109761ecac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d\" (UID: \"621bb7f9-3d3b-47df-8a4f-95109761ecac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:45.833665 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.833628    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"]
Oct 30 21:33:45.837936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.837904    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xbx\" (UniqueName: \"kubernetes.io/projected/621bb7f9-3d3b-47df-8a4f-95109761ecac-kube-api-access-29xbx\") pod \"payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d\" (UID: \"621bb7f9-3d3b-47df-8a4f-95109761ecac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:45.905044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.905006    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e38f60-dockercfg-gwt57\""
Oct 30 21:33:45.916013 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.915984    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj29t\" (UniqueName: \"kubernetes.io/projected/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-kube-api-access-dj29t\") pod \"payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c\" (UID: \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:45.916234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.916213    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c\" (UID: \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:45.989354 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.989315    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0a790f-42eb-4a41-aae9-718a87ef805f" path="/var/lib/kubelet/pods/3b0a790f-42eb-4a41-aae9-718a87ef805f/volumes"
Oct 30 21:33:45.989765 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:45.989750    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c" path="/var/lib/kubelet/pods/953c1b4e-ad15-4fbb-a1bd-c9b5aa41f90c/volumes"
Oct 30 21:33:46.011868 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:46.011834    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:46.017302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:46.017277    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c\" (UID: \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:46.017437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:46.017423    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj29t\" (UniqueName: \"kubernetes.io/projected/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-kube-api-access-dj29t\") pod \"payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c\" (UID: \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:46.018134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:46.018110    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c\" (UID: \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:46.035958 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:46.035928    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj29t\" (UniqueName: \"kubernetes.io/projected/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-kube-api-access-dj29t\") pod \"payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c\" (UID: \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:46.122892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:46.122858    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:46.327904 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:46.327877    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"]
Oct 30 21:33:46.330663 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:33:46.330633    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621bb7f9_3d3b_47df_8a4f_95109761ecac.slice/crio-92992efd8eadc7754ca6b234711e25f292497fe4ce21da9d50c948dc0225f18b WatchSource:0}: Error finding container 92992efd8eadc7754ca6b234711e25f292497fe4ce21da9d50c948dc0225f18b: Status 404 returned error can't find the container with id 92992efd8eadc7754ca6b234711e25f292497fe4ce21da9d50c948dc0225f18b
Oct 30 21:33:46.433390 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:46.433367    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"]
Oct 30 21:33:46.436062 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:33:46.436027    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eadd209_1cbc_4591_90a6_ff5082ad5cb4.slice/crio-21421a41c29ce91cec8c96ae6fdf2c4f16d54afb37814624b2c0ab7cc2774724 WatchSource:0}: Error finding container 21421a41c29ce91cec8c96ae6fdf2c4f16d54afb37814624b2c0ab7cc2774724: Status 404 returned error can't find the container with id 21421a41c29ce91cec8c96ae6fdf2c4f16d54afb37814624b2c0ab7cc2774724
Oct 30 21:33:47.268436 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:47.268397    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c" event={"ID":"9eadd209-1cbc-4591-90a6-ff5082ad5cb4","Type":"ContainerStarted","Data":"c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7"}
Oct 30 21:33:47.268949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:47.268448    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c" event={"ID":"9eadd209-1cbc-4591-90a6-ff5082ad5cb4","Type":"ContainerStarted","Data":"21421a41c29ce91cec8c96ae6fdf2c4f16d54afb37814624b2c0ab7cc2774724"}
Oct 30 21:33:47.270293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:47.270245    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d" event={"ID":"621bb7f9-3d3b-47df-8a4f-95109761ecac","Type":"ContainerStarted","Data":"b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7"}
Oct 30 21:33:47.270437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:47.270301    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d" event={"ID":"621bb7f9-3d3b-47df-8a4f-95109761ecac","Type":"ContainerStarted","Data":"92992efd8eadc7754ca6b234711e25f292497fe4ce21da9d50c948dc0225f18b"}
Oct 30 21:33:47.270437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:47.270330    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:47.307121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:47.307058    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c" podStartSLOduration=2.307040031 podStartE2EDuration="2.307040031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:47.303053297 +0000 UTC m=+1427.432304756" watchObservedRunningTime="2026-10-30 21:33:47.307040031 +0000 UTC m=+1427.436291489"
Oct 30 21:33:47.341545 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:47.341499    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d" podStartSLOduration=2.341483106 podStartE2EDuration="2.341483106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:47.338085802 +0000 UTC m=+1427.467337260" watchObservedRunningTime="2026-10-30 21:33:47.341483106 +0000 UTC m=+1427.470734581"
Oct 30 21:33:48.281560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:48.281525    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:33:49.196903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:49.196870    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:33:55.250490 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:55.250461    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:33:56.717000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.716963    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"]
Oct 30 21:33:56.721703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.721665    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.734866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.734844    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-38e5bf-serving-cert\""
Oct 30 21:33:56.749908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.749879    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"]
Oct 30 21:33:56.869778 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.869739    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkfk\" (UniqueName: \"kubernetes.io/projected/b363130d-98c5-43ca-8dbf-40c05a5bc963-kube-api-access-sqkfk\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.869968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.869935    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.870035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.870008    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b363130d-98c5-43ca-8dbf-40c05a5bc963-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.870124 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.870108    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-metrics-certs\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.971234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.971125    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkfk\" (UniqueName: \"kubernetes.io/projected/b363130d-98c5-43ca-8dbf-40c05a5bc963-kube-api-access-sqkfk\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.971407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.971279    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.971407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.971339    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b363130d-98c5-43ca-8dbf-40c05a5bc963-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.971541 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.971459    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-metrics-certs\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.972115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.972089    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b363130d-98c5-43ca-8dbf-40c05a5bc963-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.974568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.974525    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-metrics-certs\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.974659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.974587    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-maas-api-tls\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:56.991526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:56.991491    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkfk\" (UniqueName: \"kubernetes.io/projected/b363130d-98c5-43ca-8dbf-40c05a5bc963-kube-api-access-sqkfk\") pod \"maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:57.037276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:57.037240    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:57.362212 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:57.361464    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"]
Oct 30 21:33:58.283873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:58.283843    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:33:58.373822 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:58.373782    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg" event={"ID":"b363130d-98c5-43ca-8dbf-40c05a5bc963","Type":"ContainerStarted","Data":"fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3"}
Oct 30 21:33:58.374035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:58.373840    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg" event={"ID":"b363130d-98c5-43ca-8dbf-40c05a5bc963","Type":"ContainerStarted","Data":"2c11d1babeccaa391ebcfd60b28589e718f08aaf65854f6c4995315790e47542"}
Oct 30 21:33:58.374035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:58.373977    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:33:58.412739 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:58.412676    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg" podStartSLOduration=1.947462384 podStartE2EDuration="2.412663051s" totalImagesPullingTime="465.200667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:58.409787471 +0000 UTC m=+1438.539038938" watchObservedRunningTime="2026-10-30 21:33:58.412663051 +0000 UTC m=+1438.541914553"
Oct 30 21:33:59.291834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:33:59.291801    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:34:04.394025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:04.393994    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:34:08.137289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.137237    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"]
Oct 30 21:34:08.143093 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.143070    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.156701 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.156681    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b0e991-serving-cert\""
Oct 30 21:34:08.161376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.161344    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"]
Oct 30 21:34:08.232967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.232932    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.233131 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.232985    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.233227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.233201    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.233270 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.233239    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxx77\" (UniqueName: \"kubernetes.io/projected/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-kube-api-access-rxx77\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.334844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.334790    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.335055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.334965    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.335055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.334997    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxx77\" (UniqueName: \"kubernetes.io/projected/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-kube-api-access-rxx77\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.335235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.335214    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.336081 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.336048    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.337817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.337795    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.337961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.337942    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.354466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.354433    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxx77\" (UniqueName: \"kubernetes.io/projected/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-kube-api-access-rxx77\") pod \"maas-api-e2e-ipp-b-b0e991-666958c99-l98rv\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.457908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.457876    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:08.762189 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:08.762069    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"]
Oct 30 21:34:08.766171 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:34:08.766127    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2757d92d_7ca1_4159_91a5_9f98b9bcbdf8.slice/crio-62ae3feca7e71c16ef07d8bd3534ac40716b94a7febc577ad3835d39db96950a WatchSource:0}: Error finding container 62ae3feca7e71c16ef07d8bd3534ac40716b94a7febc577ad3835d39db96950a: Status 404 returned error can't find the container with id 62ae3feca7e71c16ef07d8bd3534ac40716b94a7febc577ad3835d39db96950a
Oct 30 21:34:09.466484 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.466447    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv" event={"ID":"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8","Type":"ContainerStarted","Data":"020729da56e5c430b0e17b4d8a4a90910e20eca3dbef24d87725211437cf1304"}
Oct 30 21:34:09.466956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.466514    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv" event={"ID":"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8","Type":"ContainerStarted","Data":"62ae3feca7e71c16ef07d8bd3534ac40716b94a7febc577ad3835d39db96950a"}
Oct 30 21:34:09.466956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.466546    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:09.503434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.503389    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv" podStartSLOduration=1.21598634 podStartE2EDuration="1.503376355s" totalImagesPullingTime="287.390015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:09.502370823 +0000 UTC m=+1449.631622281" watchObservedRunningTime="2026-10-30 21:34:09.503376355 +0000 UTC m=+1449.632627812"
Oct 30 21:34:09.579285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.579243    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"]
Oct 30 21:34:09.579731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.579691    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n" podUID="aa3df7fb-98d8-4ea6-8e16-0a398d776d64" containerName="maas-api" containerID="cri-o://3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194" gracePeriod=30
Oct 30 21:34:09.935920 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.935890    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:34:09.961777 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.961742    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-metrics-certs\") pod \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") "
Oct 30 21:34:09.961974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.961823    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-openshift-service-ca\") pod \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") "
Oct 30 21:34:09.962055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.962037    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-kube-api-access-kgqmt\" (UniqueName: \"kubernetes.io/projected/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-kube-api-access-kgqmt\") pod \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") "
Oct 30 21:34:09.962127 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.962107    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-maas-api-tls\") pod \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\" (UID: \"aa3df7fb-98d8-4ea6-8e16-0a398d776d64\") "
Oct 30 21:34:09.962459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.962430    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-openshift-service-ca" pod "aa3df7fb-98d8-4ea6-8e16-0a398d776d64" (UID: "aa3df7fb-98d8-4ea6-8e16-0a398d776d64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:09.962730 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.962702    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:09.964585 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.964556    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-metrics-certs" pod "aa3df7fb-98d8-4ea6-8e16-0a398d776d64" (UID: "aa3df7fb-98d8-4ea6-8e16-0a398d776d64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:09.964848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.964820    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-maas-api-tls" pod "aa3df7fb-98d8-4ea6-8e16-0a398d776d64" (UID: "aa3df7fb-98d8-4ea6-8e16-0a398d776d64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:09.964999 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:09.964984    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-kube-api-access-kgqmt" pod "aa3df7fb-98d8-4ea6-8e16-0a398d776d64" (UID: "aa3df7fb-98d8-4ea6-8e16-0a398d776d64"). InnerVolumeSpecName "kube-api-access-kgqmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:10.065132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.065062    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgqmt\" (UniqueName: \"kubernetes.io/projected/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-kube-api-access-kgqmt\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:10.065132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.065100    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:10.065583 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.065553    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa3df7fb-98d8-4ea6-8e16-0a398d776d64-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:10.474693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.474655    5886 generic.go:356] "Generic (PLEG): container finished" podID="aa3df7fb-98d8-4ea6-8e16-0a398d776d64" containerID="3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194" exitCode=0
Oct 30 21:34:10.475209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.474724    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"
Oct 30 21:34:10.475209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.474749    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n" event={"ID":"aa3df7fb-98d8-4ea6-8e16-0a398d776d64","Type":"ContainerDied","Data":"3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194"}
Oct 30 21:34:10.475209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.474840    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n" event={"ID":"aa3df7fb-98d8-4ea6-8e16-0a398d776d64","Type":"ContainerDied","Data":"522ac56cce6ee02b76cd4b0ef5e325525efc33e13cf0384d68d0902aef05fcea"}
Oct 30 21:34:10.475209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.474864    5886 scope.go:122] "RemoveContainer" containerID="3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194"
Oct 30 21:34:10.487985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.487953    5886 scope.go:122] "RemoveContainer" containerID="3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194"
Oct 30 21:34:10.488285 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:10.488258    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194\": container with ID starting with 3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194 not found: ID does not exist" containerID="3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194"
Oct 30 21:34:10.488363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.488296    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194"} err="failed to get container status \"3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194\": rpc error: code = NotFound desc = could not find container \"3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194\": container with ID starting with 3f776ed79fec3a0796ba1d2414ec4a53cc5c3511cb9c96ddccfda0d1c3d16194 not found: ID does not exist"
Oct 30 21:34:10.523508 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.523479    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"]
Oct 30 21:34:10.535317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:10.535291    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-18a043-685b9dc47d-k7v4n"]
Oct 30 21:34:11.031906 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:11.031868    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:34:11.992829 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:11.992790    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3df7fb-98d8-4ea6-8e16-0a398d776d64" path="/var/lib/kubelet/pods/aa3df7fb-98d8-4ea6-8e16-0a398d776d64/volumes"
Oct 30 21:34:15.480633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:15.480601    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:34:17.175659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:17.175618    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"]
Oct 30 21:34:17.176140 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:17.175929    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy" containerID="cri-o://3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65" gracePeriod=30
Oct 30 21:34:22.464913 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.464883    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:34:22.568531 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.568496    5886 generic.go:356] "Generic (PLEG): container finished" podID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerID="3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65" exitCode=0
Oct 30 21:34:22.568705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.568551    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"
Oct 30 21:34:22.568705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.568583    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" event={"ID":"9fcc8030-8b61-4bf2-aff3-423a4cf469f7","Type":"ContainerDied","Data":"3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65"}
Oct 30 21:34:22.568705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.568655    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn" event={"ID":"9fcc8030-8b61-4bf2-aff3-423a4cf469f7","Type":"ContainerDied","Data":"350fa8d81e161110900103e5374375b7bf5cc4fb00e088e7d7358b1575a07931"}
Oct 30 21:34:22.568705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.568685    5886 scope.go:122] "RemoveContainer" containerID="3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65"
Oct 30 21:34:22.583767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.583721    5886 scope.go:122] "RemoveContainer" containerID="8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9"
Oct 30 21:34:22.598550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.597266    5886 scope.go:122] "RemoveContainer" containerID="3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65"
Oct 30 21:34:22.598550 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:22.597798    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65\": container with ID starting with 3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65 not found: ID does not exist" containerID="3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65"
Oct 30 21:34:22.598550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.597830    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65"} err="failed to get container status \"3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65\": rpc error: code = NotFound desc = could not find container \"3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65\": container with ID starting with 3c241a3b33949f15af4fbd2ff96928e5b304287e4147bcfdde10c5fbd1087d65 not found: ID does not exist"
Oct 30 21:34:22.598550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.597855    5886 scope.go:122] "RemoveContainer" containerID="8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9"
Oct 30 21:34:22.598550 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:22.598237    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9\": container with ID starting with 8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9 not found: ID does not exist" containerID="8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9"
Oct 30 21:34:22.598550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.598258    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9"} err="failed to get container status \"8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9\": rpc error: code = NotFound desc = could not find container \"8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9\": container with ID starting with 8dd8718f7fdf25f32b520c17e89850e3b05e4b537d7d43e6e9feb85d17b2d3b9 not found: ID does not exist"
Oct 30 21:34:22.635011 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.634973    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-credential-socket\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.635268 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635052    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-data\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.635268 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635131    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-kube-api-access-9kp5g\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-kube-api-access-9kp5g\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.635268 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635211    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-envoy\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.635268 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635250    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-podinfo\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.635522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635313    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-data" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:22.635522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635344    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-socket\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.635522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635375    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-token\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-token\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.635522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635326    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-credential-socket" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:22.635522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635455    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istiod-ca-cert\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.635522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635498    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-certs\") pod \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\" (UID: \"9fcc8030-8b61-4bf2-aff3-423a4cf469f7\") "
Oct 30 21:34:22.636099 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.635999    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-socket" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:22.636099 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.636078    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-certs" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:22.636299 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.636196    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.636372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.636310    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.639467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.636328    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.639467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.636458    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istiod-ca-cert" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:22.639467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.638130    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-token" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:22.639467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.638546    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-podinfo" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:22.639467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.638671    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-kube-api-access-9kp5g" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "kube-api-access-9kp5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:22.641786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.641765    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-envoy" pod "9fcc8030-8b61-4bf2-aff3-423a4cf469f7" (UID: "9fcc8030-8b61-4bf2-aff3-423a4cf469f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:22.739568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.739484    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.739568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.739515    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.739568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.739525    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kp5g\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-kube-api-access-9kp5g\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.739568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.739533    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.739568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.739542    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.739568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.739550    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fcc8030-8b61-4bf2-aff3-423a4cf469f7-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:22.918540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.918509    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"]
Oct 30 21:34:22.930109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:22.930077    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-18a043-openshift-default-b54c7ffc5-9gnhn"]
Oct 30 21:34:23.993123 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:23.993082    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" path="/var/lib/kubelet/pods/9fcc8030-8b61-4bf2-aff3-423a4cf469f7/volumes"
Oct 30 21:34:29.022035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.021998    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"]
Oct 30 21:34:29.022589 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.022218    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz" podUID="0b8542a0-30ea-4343-a9a3-9c9c3567709f" containerName="maas-api" containerID="cri-o://ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c" gracePeriod=30
Oct 30 21:34:29.241229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.241195    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"]
Oct 30 21:34:29.241661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.241630    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9" podUID="4cd8e513-5ee6-46a9-84f7-1ef53499be33" containerName="payload-processing" containerID="cri-o://db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89" gracePeriod=30
Oct 30 21:34:29.258040 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.258006    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"]
Oct 30 21:34:29.258325 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.258300    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5" podUID="a18994f6-014f-4afd-8f50-fc672581c6ce" containerName="payload-pre-processing" containerID="cri-o://8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51" gracePeriod=30
Oct 30 21:34:29.420063 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.420035    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:34:29.437371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.436045    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8542a0-30ea-4343-a9a3-9c9c3567709f-kube-api-access-zc7wc\" (UniqueName: \"kubernetes.io/projected/0b8542a0-30ea-4343-a9a3-9c9c3567709f-kube-api-access-zc7wc\") pod \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") "
Oct 30 21:34:29.437371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.436134    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8542a0-30ea-4343-a9a3-9c9c3567709f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8542a0-30ea-4343-a9a3-9c9c3567709f-openshift-service-ca\") pod \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") "
Oct 30 21:34:29.437371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.436213    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-maas-api-tls\") pod \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") "
Oct 30 21:34:29.437371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.436289    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-metrics-certs\") pod \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\" (UID: \"0b8542a0-30ea-4343-a9a3-9c9c3567709f\") "
Oct 30 21:34:29.437752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.437720    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8542a0-30ea-4343-a9a3-9c9c3567709f-openshift-service-ca" pod "0b8542a0-30ea-4343-a9a3-9c9c3567709f" (UID: "0b8542a0-30ea-4343-a9a3-9c9c3567709f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:29.440197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.439824    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-maas-api-tls" pod "0b8542a0-30ea-4343-a9a3-9c9c3567709f" (UID: "0b8542a0-30ea-4343-a9a3-9c9c3567709f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:29.440197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.439943    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8542a0-30ea-4343-a9a3-9c9c3567709f-kube-api-access-zc7wc" pod "0b8542a0-30ea-4343-a9a3-9c9c3567709f" (UID: "0b8542a0-30ea-4343-a9a3-9c9c3567709f"). InnerVolumeSpecName "kube-api-access-zc7wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:29.440197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.440020    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-metrics-certs" pod "0b8542a0-30ea-4343-a9a3-9c9c3567709f" (UID: "0b8542a0-30ea-4343-a9a3-9c9c3567709f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:29.545183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.544331    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc7wc\" (UniqueName: \"kubernetes.io/projected/0b8542a0-30ea-4343-a9a3-9c9c3567709f-kube-api-access-zc7wc\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:29.545183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.544371    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b8542a0-30ea-4343-a9a3-9c9c3567709f-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:29.545183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.544388    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:29.545183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.544403    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8542a0-30ea-4343-a9a3-9c9c3567709f-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:29.560780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.560755    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:34:29.599208 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.599178    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:34:29.645670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.645607    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd8e513-5ee6-46a9-84f7-1ef53499be33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd8e513-5ee6-46a9-84f7-1ef53499be33-plugins-config-volume\") pod \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\" (UID: \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\") "
Oct 30 21:34:29.645889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.645693    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd8e513-5ee6-46a9-84f7-1ef53499be33-kube-api-access-lc76r\" (UniqueName: \"kubernetes.io/projected/4cd8e513-5ee6-46a9-84f7-1ef53499be33-kube-api-access-lc76r\") pod \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\" (UID: \"4cd8e513-5ee6-46a9-84f7-1ef53499be33\") "
Oct 30 21:34:29.645889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.645794    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18994f6-014f-4afd-8f50-fc672581c6ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18994f6-014f-4afd-8f50-fc672581c6ce-plugins-config-volume\") pod \"a18994f6-014f-4afd-8f50-fc672581c6ce\" (UID: \"a18994f6-014f-4afd-8f50-fc672581c6ce\") "
Oct 30 21:34:29.646061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.645914    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18994f6-014f-4afd-8f50-fc672581c6ce-kube-api-access-dkwjk\" (UniqueName: \"kubernetes.io/projected/a18994f6-014f-4afd-8f50-fc672581c6ce-kube-api-access-dkwjk\") pod \"a18994f6-014f-4afd-8f50-fc672581c6ce\" (UID: \"a18994f6-014f-4afd-8f50-fc672581c6ce\") "
Oct 30 21:34:29.648003 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.647969    5886 generic.go:356] "Generic (PLEG): container finished" podID="4cd8e513-5ee6-46a9-84f7-1ef53499be33" containerID="db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89" exitCode=0
Oct 30 21:34:29.648305 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.648282    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9" event={"ID":"4cd8e513-5ee6-46a9-84f7-1ef53499be33","Type":"ContainerDied","Data":"db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89"}
Oct 30 21:34:29.648518 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.648500    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9" event={"ID":"4cd8e513-5ee6-46a9-84f7-1ef53499be33","Type":"ContainerDied","Data":"34f42a9a8d3a226dd3dd8c1ca44afe9b38d6bed25423347791256ea3348018fb"}
Oct 30 21:34:29.648656 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.648642    5886 scope.go:122] "RemoveContainer" containerID="db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89"
Oct 30 21:34:29.649505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.649487    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"
Oct 30 21:34:29.649760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.649730    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd8e513-5ee6-46a9-84f7-1ef53499be33-plugins-config-volume" pod "4cd8e513-5ee6-46a9-84f7-1ef53499be33" (UID: "4cd8e513-5ee6-46a9-84f7-1ef53499be33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:29.650216 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.650191    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18994f6-014f-4afd-8f50-fc672581c6ce-plugins-config-volume" pod "a18994f6-014f-4afd-8f50-fc672581c6ce" (UID: "a18994f6-014f-4afd-8f50-fc672581c6ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:29.652122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.651401    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18994f6-014f-4afd-8f50-fc672581c6ce-kube-api-access-dkwjk" pod "a18994f6-014f-4afd-8f50-fc672581c6ce" (UID: "a18994f6-014f-4afd-8f50-fc672581c6ce"). InnerVolumeSpecName "kube-api-access-dkwjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:29.652122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.651492    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd8e513-5ee6-46a9-84f7-1ef53499be33-kube-api-access-lc76r" pod "4cd8e513-5ee6-46a9-84f7-1ef53499be33" (UID: "4cd8e513-5ee6-46a9-84f7-1ef53499be33"). InnerVolumeSpecName "kube-api-access-lc76r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:29.652930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.652726    5886 generic.go:356] "Generic (PLEG): container finished" podID="a18994f6-014f-4afd-8f50-fc672581c6ce" containerID="8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51" exitCode=0
Oct 30 21:34:29.652930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.652767    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5" event={"ID":"a18994f6-014f-4afd-8f50-fc672581c6ce","Type":"ContainerDied","Data":"8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51"}
Oct 30 21:34:29.652930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.652788    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"
Oct 30 21:34:29.652930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.652822    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5" event={"ID":"a18994f6-014f-4afd-8f50-fc672581c6ce","Type":"ContainerDied","Data":"ccb231939a4e8abc9125bb63637b5de0d63dd554f096f127b98ee4d65a0286ec"}
Oct 30 21:34:29.656201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.655008    5886 generic.go:356] "Generic (PLEG): container finished" podID="0b8542a0-30ea-4343-a9a3-9c9c3567709f" containerID="ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c" exitCode=0
Oct 30 21:34:29.656201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.655064    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz" event={"ID":"0b8542a0-30ea-4343-a9a3-9c9c3567709f","Type":"ContainerDied","Data":"ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c"}
Oct 30 21:34:29.656201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.655112    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz" event={"ID":"0b8542a0-30ea-4343-a9a3-9c9c3567709f","Type":"ContainerDied","Data":"86e30e28de1005979cd684fa1966a7aeab847436d139d33424be955ec7792b8e"}
Oct 30 21:34:29.656201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.655215    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"
Oct 30 21:34:29.667073 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.667027    5886 scope.go:122] "RemoveContainer" containerID="db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89"
Oct 30 21:34:29.668448 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:29.668416    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89\": container with ID starting with db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89 not found: ID does not exist" containerID="db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89"
Oct 30 21:34:29.668551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.668460    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89"} err="failed to get container status \"db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89\": rpc error: code = NotFound desc = could not find container \"db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89\": container with ID starting with db49ee6531fa1fe9a6df4a96797933cf17b34fb7da309f9772d09a64cf61dd89 not found: ID does not exist"
Oct 30 21:34:29.668716 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.668701    5886 scope.go:122] "RemoveContainer" containerID="8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51"
Oct 30 21:34:29.680133 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.680110    5886 scope.go:122] "RemoveContainer" containerID="8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51"
Oct 30 21:34:29.680613 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:29.680583    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51\": container with ID starting with 8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51 not found: ID does not exist" containerID="8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51"
Oct 30 21:34:29.680802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.680764    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51"} err="failed to get container status \"8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51\": rpc error: code = NotFound desc = could not find container \"8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51\": container with ID starting with 8b3409f382bf8df4b284beebd1cd8e0bea1b2c3734687cf649e6beee92ae1d51 not found: ID does not exist"
Oct 30 21:34:29.680802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.680799    5886 scope.go:122] "RemoveContainer" containerID="ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c"
Oct 30 21:34:29.694426 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.694396    5886 scope.go:122] "RemoveContainer" containerID="ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c"
Oct 30 21:34:29.694751 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:29.694715    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c\": container with ID starting with ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c not found: ID does not exist" containerID="ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c"
Oct 30 21:34:29.694849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.694754    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c"} err="failed to get container status \"ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c\": rpc error: code = NotFound desc = could not find container \"ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c\": container with ID starting with ddf3a616d1e7608d6f93ab5cbffe885083c720a6ff878c4a6bb20898d504b33c not found: ID does not exist"
Oct 30 21:34:29.726953 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.726921    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"]
Oct 30 21:34:29.731110 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.731082    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-229795-5d5f879d4c-vclx5"]
Oct 30 21:34:29.748424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.748394    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkwjk\" (UniqueName: \"kubernetes.io/projected/a18994f6-014f-4afd-8f50-fc672581c6ce-kube-api-access-dkwjk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:29.748424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.748421    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd8e513-5ee6-46a9-84f7-1ef53499be33-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:29.748631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.748431    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc76r\" (UniqueName: \"kubernetes.io/projected/4cd8e513-5ee6-46a9-84f7-1ef53499be33-kube-api-access-lc76r\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:29.748631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.748441    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18994f6-014f-4afd-8f50-fc672581c6ce-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:29.762034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.761999    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"]
Oct 30 21:34:29.766684 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.766177    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-229795-5b5977cdd8-b64dz"]
Oct 30 21:34:29.989367 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.989336    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8542a0-30ea-4343-a9a3-9c9c3567709f" path="/var/lib/kubelet/pods/0b8542a0-30ea-4343-a9a3-9c9c3567709f/volumes"
Oct 30 21:34:29.989902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.989885    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18994f6-014f-4afd-8f50-fc672581c6ce" path="/var/lib/kubelet/pods/a18994f6-014f-4afd-8f50-fc672581c6ce/volumes"
Oct 30 21:34:29.996641 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:29.996614    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"]
Oct 30 21:34:30.001658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:30.001636    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-229795-6cfb8b9bc4-5zzc9"]
Oct 30 21:34:31.990025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:31.989985    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd8e513-5ee6-46a9-84f7-1ef53499be33" path="/var/lib/kubelet/pods/4cd8e513-5ee6-46a9-84f7-1ef53499be33/volumes"
Oct 30 21:34:37.919928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:37.919887    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"]
Oct 30 21:34:37.920494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:37.920174    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7" podUID="6fb23ae2-c8c6-4dce-ad29-682697b0851b" containerName="payload-pre-processing" containerID="cri-o://687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c" gracePeriod=30
Oct 30 21:34:38.231346 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.231308    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:34:38.362592 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.362558    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb23ae2-c8c6-4dce-ad29-682697b0851b-kube-api-access-vhklk\" (UniqueName: \"kubernetes.io/projected/6fb23ae2-c8c6-4dce-ad29-682697b0851b-kube-api-access-vhklk\") pod \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\" (UID: \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\") "
Oct 30 21:34:38.362810 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.362789    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fb23ae2-c8c6-4dce-ad29-682697b0851b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb23ae2-c8c6-4dce-ad29-682697b0851b-plugins-config-volume\") pod \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\" (UID: \"6fb23ae2-c8c6-4dce-ad29-682697b0851b\") "
Oct 30 21:34:38.363272 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.363238    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb23ae2-c8c6-4dce-ad29-682697b0851b-plugins-config-volume" pod "6fb23ae2-c8c6-4dce-ad29-682697b0851b" (UID: "6fb23ae2-c8c6-4dce-ad29-682697b0851b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:38.364840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.364813    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb23ae2-c8c6-4dce-ad29-682697b0851b-kube-api-access-vhklk" pod "6fb23ae2-c8c6-4dce-ad29-682697b0851b" (UID: "6fb23ae2-c8c6-4dce-ad29-682697b0851b"). InnerVolumeSpecName "kube-api-access-vhklk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:38.464608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.464513    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb23ae2-c8c6-4dce-ad29-682697b0851b-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:38.464608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.464548    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhklk\" (UniqueName: \"kubernetes.io/projected/6fb23ae2-c8c6-4dce-ad29-682697b0851b-kube-api-access-vhklk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:38.730317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.730231    5886 generic.go:356] "Generic (PLEG): container finished" podID="6fb23ae2-c8c6-4dce-ad29-682697b0851b" containerID="687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c" exitCode=0
Oct 30 21:34:38.730317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.730275    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"
Oct 30 21:34:38.730317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.730305    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7" event={"ID":"6fb23ae2-c8c6-4dce-ad29-682697b0851b","Type":"ContainerDied","Data":"687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c"}
Oct 30 21:34:38.730590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.730356    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7" event={"ID":"6fb23ae2-c8c6-4dce-ad29-682697b0851b","Type":"ContainerDied","Data":"a7280cb555695c04f2f6f2e71dcd0635524f7a9c0ea58e0d78ce29dc6b01eea0"}
Oct 30 21:34:38.730590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.730380    5886 scope.go:122] "RemoveContainer" containerID="687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c"
Oct 30 21:34:38.741624 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.741605    5886 scope.go:122] "RemoveContainer" containerID="687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c"
Oct 30 21:34:38.741897 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:38.741878    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c\": container with ID starting with 687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c not found: ID does not exist" containerID="687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c"
Oct 30 21:34:38.741972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.741904    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c"} err="failed to get container status \"687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c\": rpc error: code = NotFound desc = could not find container \"687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c\": container with ID starting with 687e7d847c9be90f58a6695f7aa102f2086d0d3585bf8e95df699ddcbacccf0c not found: ID does not exist"
Oct 30 21:34:38.780781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.780751    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"]
Oct 30 21:34:38.791947 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:38.791920    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c027ae-7c96698b99-rvqj7"]
Oct 30 21:34:39.990072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:39.990039    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fb23ae2-c8c6-4dce-ad29-682697b0851b" path="/var/lib/kubelet/pods/6fb23ae2-c8c6-4dce-ad29-682697b0851b/volumes"
Oct 30 21:34:41.029220 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:41.029182    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:34:42.124451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:42.124406    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"]
Oct 30 21:34:47.125768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.125729    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc" podUID="02f28f61-0c8b-411a-8aaa-f4a2b84511eb" containerName="payload-pre-processing" containerID="cri-o://62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52" gracePeriod=25
Oct 30 21:34:47.431599 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.431566    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:34:47.590696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.590653    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-plugins-config-volume\") pod \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\" (UID: \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\") "
Oct 30 21:34:47.590961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.590937    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-kube-api-access-cknhf\" (UniqueName: \"kubernetes.io/projected/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-kube-api-access-cknhf\") pod \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\" (UID: \"02f28f61-0c8b-411a-8aaa-f4a2b84511eb\") "
Oct 30 21:34:47.592197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.592107    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-plugins-config-volume" pod "02f28f61-0c8b-411a-8aaa-f4a2b84511eb" (UID: "02f28f61-0c8b-411a-8aaa-f4a2b84511eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:47.601205 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.594268    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-kube-api-access-cknhf" pod "02f28f61-0c8b-411a-8aaa-f4a2b84511eb" (UID: "02f28f61-0c8b-411a-8aaa-f4a2b84511eb"). InnerVolumeSpecName "kube-api-access-cknhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:47.692841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.692744    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cknhf\" (UniqueName: \"kubernetes.io/projected/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-kube-api-access-cknhf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:47.692841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.692787    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02f28f61-0c8b-411a-8aaa-f4a2b84511eb-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:47.799342 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.799303    5886 generic.go:356] "Generic (PLEG): container finished" podID="02f28f61-0c8b-411a-8aaa-f4a2b84511eb" containerID="62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52" exitCode=0
Oct 30 21:34:47.799534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.799367    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"
Oct 30 21:34:47.799534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.799396    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc" event={"ID":"02f28f61-0c8b-411a-8aaa-f4a2b84511eb","Type":"ContainerDied","Data":"62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52"}
Oct 30 21:34:47.799534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.799449    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc" event={"ID":"02f28f61-0c8b-411a-8aaa-f4a2b84511eb","Type":"ContainerDied","Data":"b5b9952955ddb9c9c2d958637f69bd95d24a19400d5f0da86cab5a22c78a0207"}
Oct 30 21:34:47.799534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.799472    5886 scope.go:122] "RemoveContainer" containerID="62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52"
Oct 30 21:34:47.816779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.816760    5886 scope.go:122] "RemoveContainer" containerID="62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52"
Oct 30 21:34:47.817047 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:47.817026    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52\": container with ID starting with 62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52 not found: ID does not exist" containerID="62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52"
Oct 30 21:34:47.817099 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.817053    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52"} err="failed to get container status \"62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52\": rpc error: code = NotFound desc = could not find container \"62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52\": container with ID starting with 62ff1ca219336555a7cd4410dfc50446ba4cf15ff162ffb1844f7e7b45f99b52 not found: ID does not exist"
Oct 30 21:34:47.849066 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.849038    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"]
Oct 30 21:34:47.859724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.859699    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0e991-665c7fd8c-qbvvc"]
Oct 30 21:34:47.990061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:47.989988    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f28f61-0c8b-411a-8aaa-f4a2b84511eb" path="/var/lib/kubelet/pods/02f28f61-0c8b-411a-8aaa-f4a2b84511eb/volumes"
Oct 30 21:34:54.890412 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:54.890379    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"]
Oct 30 21:34:54.890850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:54.890620    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp" podUID="6628b759-f9c9-4d04-a472-2bbf88be896b" containerName="payload-processing" containerID="cri-o://2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec" gracePeriod=30
Oct 30 21:34:55.197411 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.197351    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:34:55.289566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.289529    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6628b759-f9c9-4d04-a472-2bbf88be896b-kube-api-access-j9tdm\" (UniqueName: \"kubernetes.io/projected/6628b759-f9c9-4d04-a472-2bbf88be896b-kube-api-access-j9tdm\") pod \"6628b759-f9c9-4d04-a472-2bbf88be896b\" (UID: \"6628b759-f9c9-4d04-a472-2bbf88be896b\") "
Oct 30 21:34:55.289762 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.289622    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6628b759-f9c9-4d04-a472-2bbf88be896b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6628b759-f9c9-4d04-a472-2bbf88be896b-plugins-config-volume\") pod \"6628b759-f9c9-4d04-a472-2bbf88be896b\" (UID: \"6628b759-f9c9-4d04-a472-2bbf88be896b\") "
Oct 30 21:34:55.290516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.290458    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6628b759-f9c9-4d04-a472-2bbf88be896b-plugins-config-volume" pod "6628b759-f9c9-4d04-a472-2bbf88be896b" (UID: "6628b759-f9c9-4d04-a472-2bbf88be896b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:55.293539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.293510    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6628b759-f9c9-4d04-a472-2bbf88be896b-kube-api-access-j9tdm" pod "6628b759-f9c9-4d04-a472-2bbf88be896b" (UID: "6628b759-f9c9-4d04-a472-2bbf88be896b"). InnerVolumeSpecName "kube-api-access-j9tdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:55.390866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.390831    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9tdm\" (UniqueName: \"kubernetes.io/projected/6628b759-f9c9-4d04-a472-2bbf88be896b-kube-api-access-j9tdm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:55.390866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.390863    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6628b759-f9c9-4d04-a472-2bbf88be896b-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:34:55.866881 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.866848    5886 generic.go:356] "Generic (PLEG): container finished" podID="6628b759-f9c9-4d04-a472-2bbf88be896b" containerID="2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec" exitCode=0
Oct 30 21:34:55.867162 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.866933    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"
Oct 30 21:34:55.867307 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.866976    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp" event={"ID":"6628b759-f9c9-4d04-a472-2bbf88be896b","Type":"ContainerDied","Data":"2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec"}
Oct 30 21:34:55.867368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.867347    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp" event={"ID":"6628b759-f9c9-4d04-a472-2bbf88be896b","Type":"ContainerDied","Data":"08e6b503ec6c470ea9d33b73d5a6ec40cdc5cd7dc76d07dfabefb55deb67507f"}
Oct 30 21:34:55.867406 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.867386    5886 scope.go:122] "RemoveContainer" containerID="2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec"
Oct 30 21:34:55.879897 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.879873    5886 scope.go:122] "RemoveContainer" containerID="2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec"
Oct 30 21:34:55.880230 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:34:55.880209    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec\": container with ID starting with 2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec not found: ID does not exist" containerID="2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec"
Oct 30 21:34:55.880330 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.880236    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec"} err="failed to get container status \"2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec\": rpc error: code = NotFound desc = could not find container \"2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec\": container with ID starting with 2887f1fe47b5364c73468292b0ff6a51e9a5f7ab027f2eb1420d5a1c405a8dec not found: ID does not exist"
Oct 30 21:34:55.921249 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.921218    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"]
Oct 30 21:34:55.928701 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.928673    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7222e2-784768fbd-8f8xp"]
Oct 30 21:34:55.989497 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:34:55.989467    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6628b759-f9c9-4d04-a472-2bbf88be896b" path="/var/lib/kubelet/pods/6628b759-f9c9-4d04-a472-2bbf88be896b/volumes"
Oct 30 21:35:01.677347 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.677309    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"]
Oct 30 21:35:01.677812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.677634    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l" podUID="56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" containerName="maas-api" containerID="cri-o://4737076ba518caf69b3f5f6052859be1efa503b905a2d9661d2cd55634ccf9a0" gracePeriod=30
Oct 30 21:35:01.784172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.784118    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l" podUID="56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": dial tcp 10.133.0.107:8443: connect: connection refused"
Oct 30 21:35:01.822998 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.822958    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"]
Oct 30 21:35:01.823291 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.823242    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c" podUID="9eadd209-1cbc-4591-90a6-ff5082ad5cb4" containerName="payload-processing" containerID="cri-o://c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7" gracePeriod=30
Oct 30 21:35:01.845721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.845431    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"]
Oct 30 21:35:01.845812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.845744    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d" podUID="621bb7f9-3d3b-47df-8a4f-95109761ecac" containerName="payload-pre-processing" containerID="cri-o://b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7" gracePeriod=30
Oct 30 21:35:01.928118 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.925498    5886 generic.go:356] "Generic (PLEG): container finished" podID="56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" containerID="4737076ba518caf69b3f5f6052859be1efa503b905a2d9661d2cd55634ccf9a0" exitCode=0
Oct 30 21:35:01.928118 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:01.925647    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l" event={"ID":"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f","Type":"ContainerDied","Data":"4737076ba518caf69b3f5f6052859be1efa503b905a2d9661d2cd55634ccf9a0"}
Oct 30 21:35:02.076302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.072893    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:35:02.150689 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.150655    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:35:02.248424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.248378    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-maas-api-tls\") pod \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") "
Oct 30 21:35:02.251786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.251753    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-metrics-certs\") pod \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") "
Oct 30 21:35:02.252046 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.252021    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-openshift-service-ca\") pod \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") "
Oct 30 21:35:02.252245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.252218    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-plugins-config-volume\") pod \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\" (UID: \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\") "
Oct 30 21:35:02.252824 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.251667    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-maas-api-tls" pod "56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" (UID: "56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:02.252824 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.252471    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-openshift-service-ca" pod "56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" (UID: "56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:02.252824 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.252602    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-plugins-config-volume" pod "9eadd209-1cbc-4591-90a6-ff5082ad5cb4" (UID: "9eadd209-1cbc-4591-90a6-ff5082ad5cb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:02.253373 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.252753    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-kube-api-access-dj29t\" (UniqueName: \"kubernetes.io/projected/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-kube-api-access-dj29t\") pod \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\" (UID: \"9eadd209-1cbc-4591-90a6-ff5082ad5cb4\") "
Oct 30 21:35:02.253586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.253569    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-kube-api-access-qr8cl\" (UniqueName: \"kubernetes.io/projected/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-kube-api-access-qr8cl\") pod \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\" (UID: \"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f\") "
Oct 30 21:35:02.254580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.254559    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.254762 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.254746    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.254859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.254847    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.255078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.255044    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-metrics-certs" pod "56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" (UID: "56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:02.256922 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.256888    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-kube-api-access-dj29t" pod "9eadd209-1cbc-4591-90a6-ff5082ad5cb4" (UID: "9eadd209-1cbc-4591-90a6-ff5082ad5cb4"). InnerVolumeSpecName "kube-api-access-dj29t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.259520 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.259481    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-kube-api-access-qr8cl" pod "56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" (UID: "56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f"). InnerVolumeSpecName "kube-api-access-qr8cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.286095 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.286032    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:35:02.355534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.355438    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621bb7f9-3d3b-47df-8a4f-95109761ecac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621bb7f9-3d3b-47df-8a4f-95109761ecac-plugins-config-volume\") pod \"621bb7f9-3d3b-47df-8a4f-95109761ecac\" (UID: \"621bb7f9-3d3b-47df-8a4f-95109761ecac\") "
Oct 30 21:35:02.355908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.355887    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621bb7f9-3d3b-47df-8a4f-95109761ecac-kube-api-access-29xbx\" (UniqueName: \"kubernetes.io/projected/621bb7f9-3d3b-47df-8a4f-95109761ecac-kube-api-access-29xbx\") pod \"621bb7f9-3d3b-47df-8a4f-95109761ecac\" (UID: \"621bb7f9-3d3b-47df-8a4f-95109761ecac\") "
Oct 30 21:35:02.356642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.356619    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr8cl\" (UniqueName: \"kubernetes.io/projected/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-kube-api-access-qr8cl\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.356790 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.356776    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.356881 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.356870    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj29t\" (UniqueName: \"kubernetes.io/projected/9eadd209-1cbc-4591-90a6-ff5082ad5cb4-kube-api-access-dj29t\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.357411 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.357382    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621bb7f9-3d3b-47df-8a4f-95109761ecac-plugins-config-volume" pod "621bb7f9-3d3b-47df-8a4f-95109761ecac" (UID: "621bb7f9-3d3b-47df-8a4f-95109761ecac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:02.358992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.358970    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621bb7f9-3d3b-47df-8a4f-95109761ecac-kube-api-access-29xbx" pod "621bb7f9-3d3b-47df-8a4f-95109761ecac" (UID: "621bb7f9-3d3b-47df-8a4f-95109761ecac"). InnerVolumeSpecName "kube-api-access-29xbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.457835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.457795    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621bb7f9-3d3b-47df-8a4f-95109761ecac-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.458029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.457847    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29xbx\" (UniqueName: \"kubernetes.io/projected/621bb7f9-3d3b-47df-8a4f-95109761ecac-kube-api-access-29xbx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:02.934348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.934315    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"
Oct 30 21:35:02.934839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.934347    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l" event={"ID":"56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f","Type":"ContainerDied","Data":"4e4162d91026105f1f36f1feb4a765e19ac7528cad848b1c2063f9793299b24f"}
Oct 30 21:35:02.934839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.934408    5886 scope.go:122] "RemoveContainer" containerID="4737076ba518caf69b3f5f6052859be1efa503b905a2d9661d2cd55634ccf9a0"
Oct 30 21:35:02.935984 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.935954    5886 generic.go:356] "Generic (PLEG): container finished" podID="9eadd209-1cbc-4591-90a6-ff5082ad5cb4" containerID="c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7" exitCode=0
Oct 30 21:35:02.936122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.935988    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c" event={"ID":"9eadd209-1cbc-4591-90a6-ff5082ad5cb4","Type":"ContainerDied","Data":"c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7"}
Oct 30 21:35:02.936122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.936008    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"
Oct 30 21:35:02.936122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.936056    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c" event={"ID":"9eadd209-1cbc-4591-90a6-ff5082ad5cb4","Type":"ContainerDied","Data":"21421a41c29ce91cec8c96ae6fdf2c4f16d54afb37814624b2c0ab7cc2774724"}
Oct 30 21:35:02.937557 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.937536    5886 generic.go:356] "Generic (PLEG): container finished" podID="621bb7f9-3d3b-47df-8a4f-95109761ecac" containerID="b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7" exitCode=0
Oct 30 21:35:02.937681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.937584    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"
Oct 30 21:35:02.937681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.937614    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d" event={"ID":"621bb7f9-3d3b-47df-8a4f-95109761ecac","Type":"ContainerDied","Data":"b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7"}
Oct 30 21:35:02.937681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.937649    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d" event={"ID":"621bb7f9-3d3b-47df-8a4f-95109761ecac","Type":"ContainerDied","Data":"92992efd8eadc7754ca6b234711e25f292497fe4ce21da9d50c948dc0225f18b"}
Oct 30 21:35:02.947817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.947795    5886 scope.go:122] "RemoveContainer" containerID="c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7"
Oct 30 21:35:02.959292 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.959271    5886 scope.go:122] "RemoveContainer" containerID="c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7"
Oct 30 21:35:02.959589 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:35:02.959566    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7\": container with ID starting with c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7 not found: ID does not exist" containerID="c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7"
Oct 30 21:35:02.959677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.959594    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7"} err="failed to get container status \"c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7\": rpc error: code = NotFound desc = could not find container \"c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7\": container with ID starting with c9b58129e1db063c26d537ad135dbc3c1c01838713044797ed201812b8c18bc7 not found: ID does not exist"
Oct 30 21:35:02.959677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.959612    5886 scope.go:122] "RemoveContainer" containerID="b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7"
Oct 30 21:35:02.971908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.971878    5886 scope.go:122] "RemoveContainer" containerID="b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7"
Oct 30 21:35:02.972346 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:35:02.972314    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7\": container with ID starting with b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7 not found: ID does not exist" containerID="b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7"
Oct 30 21:35:02.972513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:02.972356    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7"} err="failed to get container status \"b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7\": rpc error: code = NotFound desc = could not find container \"b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7\": container with ID starting with b6cf5a848340948db2dbc538e29c090172153a0399f477d6479d4796118f65d7 not found: ID does not exist"
Oct 30 21:35:03.017915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.017828    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"]
Oct 30 21:35:03.021290 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.021263    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e38f60-5b8545d87b-rtc7c"]
Oct 30 21:35:03.060005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.059976    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"]
Oct 30 21:35:03.065123 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.065099    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e38f60-585567cccc-klx6l"]
Oct 30 21:35:03.098884 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.098856    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"]
Oct 30 21:35:03.108746 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.108711    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e38f60-665ff4b8bc-fbz6d"]
Oct 30 21:35:03.990255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.990214    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" path="/var/lib/kubelet/pods/56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f/volumes"
Oct 30 21:35:03.990652 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.990628    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621bb7f9-3d3b-47df-8a4f-95109761ecac" path="/var/lib/kubelet/pods/621bb7f9-3d3b-47df-8a4f-95109761ecac/volumes"
Oct 30 21:35:03.990986 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:03.990971    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eadd209-1cbc-4591-90a6-ff5082ad5cb4" path="/var/lib/kubelet/pods/9eadd209-1cbc-4591-90a6-ff5082ad5cb4/volumes"
Oct 30 21:35:05.281750 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.281716    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"]
Oct 30 21:35:05.282686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282593    5886 cpu_manager.go:414] "containerMap: removing container" podUID="6fb23ae2-c8c6-4dce-ad29-682697b0851b" containerName="payload-pre-processing"
Oct 30 21:35:05.282686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282619    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb23ae2-c8c6-4dce-ad29-682697b0851b" containerName="payload-pre-processing"
Oct 30 21:35:05.282686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282639    5886 cpu_manager.go:414] "containerMap: removing container" podUID="02f28f61-0c8b-411a-8aaa-f4a2b84511eb" containerName="payload-pre-processing"
Oct 30 21:35:05.282686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282648    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f28f61-0c8b-411a-8aaa-f4a2b84511eb" containerName="payload-pre-processing"
Oct 30 21:35:05.282686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282672    5886 cpu_manager.go:414] "containerMap: removing container" podUID="4cd8e513-5ee6-46a9-84f7-1ef53499be33" containerName="payload-processing"
Oct 30 21:35:05.282686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282680    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd8e513-5ee6-46a9-84f7-1ef53499be33" containerName="payload-processing"
Oct 30 21:35:05.282686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282695    5886 cpu_manager.go:414] "containerMap: removing container" podUID="9eadd209-1cbc-4591-90a6-ff5082ad5cb4" containerName="payload-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282703    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eadd209-1cbc-4591-90a6-ff5082ad5cb4" containerName="payload-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282714    5886 cpu_manager.go:414] "containerMap: removing container" podUID="aa3df7fb-98d8-4ea6-8e16-0a398d776d64" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282721    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3df7fb-98d8-4ea6-8e16-0a398d776d64" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282734    5886 cpu_manager.go:414] "containerMap: removing container" podUID="56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282749    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282761    5886 cpu_manager.go:414] "containerMap: removing container" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282768    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282794    5886 cpu_manager.go:414] "containerMap: removing container" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282802    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282814    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a18994f6-014f-4afd-8f50-fc672581c6ce" containerName="payload-pre-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282822    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18994f6-014f-4afd-8f50-fc672581c6ce" containerName="payload-pre-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282845    5886 cpu_manager.go:414] "containerMap: removing container" podUID="0b8542a0-30ea-4343-a9a3-9c9c3567709f" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282852    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8542a0-30ea-4343-a9a3-9c9c3567709f" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282877    5886 cpu_manager.go:414] "containerMap: removing container" podUID="621bb7f9-3d3b-47df-8a4f-95109761ecac" containerName="payload-pre-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282884    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621bb7f9-3d3b-47df-8a4f-95109761ecac" containerName="payload-pre-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282899    5886 cpu_manager.go:414] "containerMap: removing container" podUID="6628b759-f9c9-4d04-a472-2bbf88be896b" containerName="payload-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.282905    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6628b759-f9c9-4d04-a472-2bbf88be896b" containerName="payload-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283092    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283106    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18994f6-014f-4afd-8f50-fc672581c6ce" containerName="payload-pre-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283119    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8542a0-30ea-4343-a9a3-9c9c3567709f" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283131    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="621bb7f9-3d3b-47df-8a4f-95109761ecac" containerName="payload-pre-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283141    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="6628b759-f9c9-4d04-a472-2bbf88be896b" containerName="payload-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283151    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb23ae2-c8c6-4dce-ad29-682697b0851b" containerName="payload-pre-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283173    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f28f61-0c8b-411a-8aaa-f4a2b84511eb" containerName="payload-pre-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283184    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd8e513-5ee6-46a9-84f7-1ef53499be33" containerName="payload-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283193    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eadd209-1cbc-4591-90a6-ff5082ad5cb4" containerName="payload-processing"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283202    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3df7fb-98d8-4ea6-8e16-0a398d776d64" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283212    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f6e9d2-1bb2-4b2c-bbee-95bc89183d5f" containerName="maas-api"
Oct 30 21:35:05.283313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.283221    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcc8030-8b61-4bf2-aff3-423a4cf469f7" containerName="istio-proxy"
Oct 30 21:35:05.288665 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.288641    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.307516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.307466    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3a5c92-openshift-default-dockercfg-qvsjt\""
Oct 30 21:35:05.322120 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.322089    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"]
Oct 30 21:35:05.398238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398204    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-envoy\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.398410 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398375    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-certs\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.398477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398453    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istiod-ca-cert\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.398552 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398536    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-token\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.398598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398584    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-socket\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.398629 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398619    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88db8\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-kube-api-access-88db8\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.398700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398685    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-podinfo\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.398731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398711    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-credential-socket\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.398774 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.398730    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-data\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500069 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500007    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-certs\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500069 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500062    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istiod-ca-cert\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500117    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-token\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500142    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-socket\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500201    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88db8\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-kube-api-access-88db8\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500236    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-podinfo\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500262    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-credential-socket\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500289    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-data\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.500428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500333    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-envoy\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.501012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500580    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-certs\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.501012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.500928    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istiod-ca-cert\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.501151 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.501134    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-credential-socket\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.501268 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.501196    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-socket\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.501456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.501431    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-data\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.504131 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.504084    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-podinfo\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.506756 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.506717    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-envoy\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.544136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.544028    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-token\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.544732 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.544687    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88db8\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-kube-api-access-88db8\") pod \"e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") " pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.606004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.605639    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:05.939664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.939637    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"]
Oct 30 21:35:05.947203 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:35:05.947166    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16f82a5_11e5_4c70_996f_a7d5ea36573a.slice/crio-66d1af4820d80bb20215bebac8f2cfb629b0342a025821a0ff8dddaa9d34f508 WatchSource:0}: Error finding container 66d1af4820d80bb20215bebac8f2cfb629b0342a025821a0ff8dddaa9d34f508: Status 404 returned error can't find the container with id 66d1af4820d80bb20215bebac8f2cfb629b0342a025821a0ff8dddaa9d34f508
Oct 30 21:35:05.949911 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.949877    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:05.950022 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.949974    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:05.950022 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.950008    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:35:05.974107 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:05.974077    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" event={"ID":"f16f82a5-11e5-4c70-996f-a7d5ea36573a","Type":"ContainerStarted","Data":"66d1af4820d80bb20215bebac8f2cfb629b0342a025821a0ff8dddaa9d34f508"}
Oct 30 21:35:06.714823 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:06.714781    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"]
Oct 30 21:35:06.715370 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:06.715036    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6" podUID="517c9e29-f434-496f-ad7d-8650e88be247" containerName="maas-api" containerID="cri-o://c187eb212c063c446d54b48f721b0c3e5574b8ea28949401b0ac79bdfd8ad2c1" gracePeriod=30
Oct 30 21:35:07.016234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.016148    5886 generic.go:356] "Generic (PLEG): container finished" podID="517c9e29-f434-496f-ad7d-8650e88be247" containerID="c187eb212c063c446d54b48f721b0c3e5574b8ea28949401b0ac79bdfd8ad2c1" exitCode=0
Oct 30 21:35:07.016470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.016225    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6" event={"ID":"517c9e29-f434-496f-ad7d-8650e88be247","Type":"ContainerDied","Data":"c187eb212c063c446d54b48f721b0c3e5574b8ea28949401b0ac79bdfd8ad2c1"}
Oct 30 21:35:07.019560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.019506    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" event={"ID":"f16f82a5-11e5-4c70-996f-a7d5ea36573a","Type":"ContainerStarted","Data":"99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e"}
Oct 30 21:35:07.035444 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.035403    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:35:07.075191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.071759    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podStartSLOduration=2.071736833 podStartE2EDuration="2.071736833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:07.068850714 +0000 UTC m=+1507.198102171" watchObservedRunningTime="2026-10-30 21:35:07.071736833 +0000 UTC m=+1507.200988290"
Oct 30 21:35:07.139393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.139349    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-metrics-certs\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-metrics-certs\") pod \"517c9e29-f434-496f-ad7d-8650e88be247\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") "
Oct 30 21:35:07.139586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.139443    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517c9e29-f434-496f-ad7d-8650e88be247-kube-api-access-qc6vv\" (UniqueName: \"kubernetes.io/projected/517c9e29-f434-496f-ad7d-8650e88be247-kube-api-access-qc6vv\") pod \"517c9e29-f434-496f-ad7d-8650e88be247\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") "
Oct 30 21:35:07.139586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.139493    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-maas-api-tls\") pod \"517c9e29-f434-496f-ad7d-8650e88be247\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") "
Oct 30 21:35:07.139586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.139523    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517c9e29-f434-496f-ad7d-8650e88be247-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c9e29-f434-496f-ad7d-8650e88be247-openshift-service-ca\") pod \"517c9e29-f434-496f-ad7d-8650e88be247\" (UID: \"517c9e29-f434-496f-ad7d-8650e88be247\") "
Oct 30 21:35:07.140201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.140169    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517c9e29-f434-496f-ad7d-8650e88be247-openshift-service-ca" pod "517c9e29-f434-496f-ad7d-8650e88be247" (UID: "517c9e29-f434-496f-ad7d-8650e88be247"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:07.142133 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.142096    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-maas-api-tls" pod "517c9e29-f434-496f-ad7d-8650e88be247" (UID: "517c9e29-f434-496f-ad7d-8650e88be247"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:07.142519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.142489    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-metrics-certs" pod "517c9e29-f434-496f-ad7d-8650e88be247" (UID: "517c9e29-f434-496f-ad7d-8650e88be247"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:07.143450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.143423    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517c9e29-f434-496f-ad7d-8650e88be247-kube-api-access-qc6vv" pod "517c9e29-f434-496f-ad7d-8650e88be247" (UID: "517c9e29-f434-496f-ad7d-8650e88be247"). InnerVolumeSpecName "kube-api-access-qc6vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:07.241387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.241308    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6vv\" (UniqueName: \"kubernetes.io/projected/517c9e29-f434-496f-ad7d-8650e88be247-kube-api-access-qc6vv\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:07.241387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.241342    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:07.241387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.241352    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517c9e29-f434-496f-ad7d-8650e88be247-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:07.241387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.241364    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517c9e29-f434-496f-ad7d-8650e88be247-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:07.606312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.606278    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:07.612989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:07.612956    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:08.026783 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:08.026753    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"
Oct 30 21:35:08.027194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:08.026780    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6" event={"ID":"517c9e29-f434-496f-ad7d-8650e88be247","Type":"ContainerDied","Data":"4e2f656c9b1644c7aa8c82b13f0a8dac6ec8e728d65997dc340201e0c85a889e"}
Oct 30 21:35:08.027194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:08.026843    5886 scope.go:122] "RemoveContainer" containerID="c187eb212c063c446d54b48f721b0c3e5574b8ea28949401b0ac79bdfd8ad2c1"
Oct 30 21:35:08.027408 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:08.027357    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:08.028429 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:08.028399    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:35:08.082014 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:08.081986    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"]
Oct 30 21:35:08.093957 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:08.093922    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1659a-6974fb4486-mthk6"]
Oct 30 21:35:10.001435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:10.001398    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517c9e29-f434-496f-ad7d-8650e88be247" path="/var/lib/kubelet/pods/517c9e29-f434-496f-ad7d-8650e88be247/volumes"
Oct 30 21:35:11.029580 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:35:11.029535    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:35:20.787625 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.787586    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"]
Oct 30 21:35:20.788265 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.788246    5886 cpu_manager.go:414] "containerMap: removing container" podUID="517c9e29-f434-496f-ad7d-8650e88be247" containerName="maas-api"
Oct 30 21:35:20.788328 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.788267    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517c9e29-f434-496f-ad7d-8650e88be247" containerName="maas-api"
Oct 30 21:35:20.788478 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.788457    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="517c9e29-f434-496f-ad7d-8650e88be247" containerName="maas-api"
Oct 30 21:35:20.791969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.791948    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:20.811058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.811026    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-13658f-dockercfg-vr5gv\""
Oct 30 21:35:20.822747 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.822710    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-13658f\""
Oct 30 21:35:20.827534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.827504    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"]
Oct 30 21:35:20.856655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.855773    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"]
Oct 30 21:35:20.878240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.869885    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:20.890326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.890227    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"]
Oct 30 21:35:20.950378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.950341    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421b5cbd-ab97-449d-8330-85d7b653982e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm\" (UID: \"421b5cbd-ab97-449d-8330-85d7b653982e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:20.950561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.950398    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxq5\" (UniqueName: \"kubernetes.io/projected/b21188ce-77a0-4112-9773-6a1fa0a1f955-kube-api-access-jkxq5\") pod \"payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg\" (UID: \"b21188ce-77a0-4112-9773-6a1fa0a1f955\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:20.950561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.950529    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgmm\" (UniqueName: \"kubernetes.io/projected/421b5cbd-ab97-449d-8330-85d7b653982e-kube-api-access-vfgmm\") pod \"payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm\" (UID: \"421b5cbd-ab97-449d-8330-85d7b653982e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:20.950916 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:20.950728    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21188ce-77a0-4112-9773-6a1fa0a1f955-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg\" (UID: \"b21188ce-77a0-4112-9773-6a1fa0a1f955\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:21.052438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.052331    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421b5cbd-ab97-449d-8330-85d7b653982e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm\" (UID: \"421b5cbd-ab97-449d-8330-85d7b653982e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:21.052438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.052382    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxq5\" (UniqueName: \"kubernetes.io/projected/b21188ce-77a0-4112-9773-6a1fa0a1f955-kube-api-access-jkxq5\") pod \"payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg\" (UID: \"b21188ce-77a0-4112-9773-6a1fa0a1f955\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:21.052748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.052537    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgmm\" (UniqueName: \"kubernetes.io/projected/421b5cbd-ab97-449d-8330-85d7b653982e-kube-api-access-vfgmm\") pod \"payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm\" (UID: \"421b5cbd-ab97-449d-8330-85d7b653982e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:21.052748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.052719    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21188ce-77a0-4112-9773-6a1fa0a1f955-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg\" (UID: \"b21188ce-77a0-4112-9773-6a1fa0a1f955\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:21.053319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.053297    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421b5cbd-ab97-449d-8330-85d7b653982e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm\" (UID: \"421b5cbd-ab97-449d-8330-85d7b653982e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:21.053502 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.053484    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21188ce-77a0-4112-9773-6a1fa0a1f955-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg\" (UID: \"b21188ce-77a0-4112-9773-6a1fa0a1f955\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:21.075013 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.074986    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxq5\" (UniqueName: \"kubernetes.io/projected/b21188ce-77a0-4112-9773-6a1fa0a1f955-kube-api-access-jkxq5\") pod \"payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg\" (UID: \"b21188ce-77a0-4112-9773-6a1fa0a1f955\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:21.075235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.075176    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgmm\" (UniqueName: \"kubernetes.io/projected/421b5cbd-ab97-449d-8330-85d7b653982e-kube-api-access-vfgmm\") pod \"payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm\" (UID: \"421b5cbd-ab97-449d-8330-85d7b653982e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:21.108566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.108534    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:21.211797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.211761    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:21.425559 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.425524    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"]
Oct 30 21:35:21.429332 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:35:21.429303    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21188ce_77a0_4112_9773_6a1fa0a1f955.slice/crio-bf87fb1d8512f03066d86a605da30b8981bbeacd979208bd6e25bf4474f3040e WatchSource:0}: Error finding container bf87fb1d8512f03066d86a605da30b8981bbeacd979208bd6e25bf4474f3040e: Status 404 returned error can't find the container with id bf87fb1d8512f03066d86a605da30b8981bbeacd979208bd6e25bf4474f3040e
Oct 30 21:35:21.546036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:21.545993    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"]
Oct 30 21:35:21.549701 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:35:21.549667    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421b5cbd_ab97_449d_8330_85d7b653982e.slice/crio-699fc7016d906904e9ffef3ad325e9316b99105001907b91b0e3c67f55279919 WatchSource:0}: Error finding container 699fc7016d906904e9ffef3ad325e9316b99105001907b91b0e3c67f55279919: Status 404 returned error can't find the container with id 699fc7016d906904e9ffef3ad325e9316b99105001907b91b0e3c67f55279919
Oct 30 21:35:22.003265 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.003232    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-6l9b4_ee881cfc-ad3c-4017-8504-722b0543c67f/payload-processing/0.log"
Oct 30 21:35:22.128199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.128135    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg" event={"ID":"b21188ce-77a0-4112-9773-6a1fa0a1f955","Type":"ContainerStarted","Data":"29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972"}
Oct 30 21:35:22.128389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.128220    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg" event={"ID":"b21188ce-77a0-4112-9773-6a1fa0a1f955","Type":"ContainerStarted","Data":"bf87fb1d8512f03066d86a605da30b8981bbeacd979208bd6e25bf4474f3040e"}
Oct 30 21:35:22.128389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.128361    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:22.129828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.129798    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm" event={"ID":"421b5cbd-ab97-449d-8330-85d7b653982e","Type":"ContainerStarted","Data":"8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715"}
Oct 30 21:35:22.129967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.129843    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm" event={"ID":"421b5cbd-ab97-449d-8330-85d7b653982e","Type":"ContainerStarted","Data":"699fc7016d906904e9ffef3ad325e9316b99105001907b91b0e3c67f55279919"}
Oct 30 21:35:22.129967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.129875    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:22.180206 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.180138    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg" podStartSLOduration=2.180124587 podStartE2EDuration="2.180124587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:22.178503009 +0000 UTC m=+1522.307754471" watchObservedRunningTime="2026-10-30 21:35:22.180124587 +0000 UTC m=+1522.309376043"
Oct 30 21:35:22.226622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.226590    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-6l9b4_ee881cfc-ad3c-4017-8504-722b0543c67f/payload-processing/0.log"
Oct 30 21:35:22.232229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:22.232177    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm" podStartSLOduration=2.232147156 podStartE2EDuration="2.232147156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:22.228705035 +0000 UTC m=+1522.357956494" watchObservedRunningTime="2026-10-30 21:35:22.232147156 +0000 UTC m=+1522.361398616"
Oct 30 21:35:25.060365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.060331    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"]
Oct 30 21:35:25.064772 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.064751    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:25.098404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.098370    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"]
Oct 30 21:35:25.108880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.108269    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-26a13f-dockercfg-r9b5h\""
Oct 30 21:35:25.164597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.164568    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-26a13f\""
Oct 30 21:35:25.218522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.218492    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b55af1b4-7a9b-4a6f-8801-481e67895f4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x\" (UID: \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:25.218693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.218530    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqkvk\" (UniqueName: \"kubernetes.io/projected/b55af1b4-7a9b-4a6f-8801-481e67895f4a-kube-api-access-lqkvk\") pod \"payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x\" (UID: \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:25.329230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.321137    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b55af1b4-7a9b-4a6f-8801-481e67895f4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x\" (UID: \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:25.329230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.327323    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqkvk\" (UniqueName: \"kubernetes.io/projected/b55af1b4-7a9b-4a6f-8801-481e67895f4a-kube-api-access-lqkvk\") pod \"payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x\" (UID: \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:25.329230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.328764    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b55af1b4-7a9b-4a6f-8801-481e67895f4a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x\" (UID: \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:25.360769 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.360735    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqkvk\" (UniqueName: \"kubernetes.io/projected/b55af1b4-7a9b-4a6f-8801-481e67895f4a-kube-api-access-lqkvk\") pod \"payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x\" (UID: \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:25.389210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.385397    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:25.717909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:25.717877    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"]
Oct 30 21:35:25.721346 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:35:25.721317    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55af1b4_7a9b_4a6f_8801_481e67895f4a.slice/crio-864816b524186d3731f2d5f7e66a590121efd604ecc4a00a1043b5ef0de4241e WatchSource:0}: Error finding container 864816b524186d3731f2d5f7e66a590121efd604ecc4a00a1043b5ef0de4241e: Status 404 returned error can't find the container with id 864816b524186d3731f2d5f7e66a590121efd604ecc4a00a1043b5ef0de4241e
Oct 30 21:35:26.164555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:26.164515    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x" event={"ID":"b55af1b4-7a9b-4a6f-8801-481e67895f4a","Type":"ContainerStarted","Data":"39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1"}
Oct 30 21:35:26.164999 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:26.164582    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x" event={"ID":"b55af1b4-7a9b-4a6f-8801-481e67895f4a","Type":"ContainerStarted","Data":"864816b524186d3731f2d5f7e66a590121efd604ecc4a00a1043b5ef0de4241e"}
Oct 30 21:35:26.164999 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:26.164736    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:26.232708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:26.232650    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x" podStartSLOduration=1.23263352 podStartE2EDuration="1.23263352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:26.229711559 +0000 UTC m=+1526.358963016" watchObservedRunningTime="2026-10-30 21:35:26.23263352 +0000 UTC m=+1526.361885015"
Oct 30 21:35:32.101179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.100451    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"]
Oct 30 21:35:32.105315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.105287    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.121214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.121125    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3a5c92-serving-cert\""
Oct 30 21:35:32.136716 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.136685    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"]
Oct 30 21:35:32.218873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.218839    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5b4\" (UniqueName: \"kubernetes.io/projected/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-kube-api-access-jh5b4\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.218873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.218874    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-metrics-certs\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.219082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.218952    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.219119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.219087    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.321392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.321350    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5b4\" (UniqueName: \"kubernetes.io/projected/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-kube-api-access-jh5b4\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.321613 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.321421    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-metrics-certs\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.321613 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.321493    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.321742 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.321683    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.322487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.322460    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.324346 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.324323    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-metrics-certs\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.324471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.324437    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.342716 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.342680    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5b4\" (UniqueName: \"kubernetes.io/projected/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-kube-api-access-jh5b4\") pod \"maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.429942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.429904    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:32.743622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:32.743539    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"]
Oct 30 21:35:32.747324 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:35:32.747294    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91f8d27_558e_4456_89d5_daf1c1c1ebe6.slice/crio-e6399bbf19e3f5538721a5f5dbd31467ba8033f37108c0e759113ae60130fe3e WatchSource:0}: Error finding container e6399bbf19e3f5538721a5f5dbd31467ba8033f37108c0e759113ae60130fe3e: Status 404 returned error can't find the container with id e6399bbf19e3f5538721a5f5dbd31467ba8033f37108c0e759113ae60130fe3e
Oct 30 21:35:33.141766 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:33.141739    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:35:33.142119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:33.141804    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:35:33.260223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:33.260185    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf" event={"ID":"e91f8d27-558e-4456-89d5-daf1c1c1ebe6","Type":"ContainerStarted","Data":"5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952"}
Oct 30 21:35:33.260417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:33.260244    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf" event={"ID":"e91f8d27-558e-4456-89d5-daf1c1c1ebe6","Type":"ContainerStarted","Data":"e6399bbf19e3f5538721a5f5dbd31467ba8033f37108c0e759113ae60130fe3e"}
Oct 30 21:35:33.260417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:33.260384    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:37.175047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:37.175010    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:35:37.229867 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:37.229806    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf" podStartSLOduration=4.866738948 podStartE2EDuration="5.22978979s" totalImagesPullingTime="363.050842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:33.311399512 +0000 UTC m=+1533.440650968" watchObservedRunningTime="2026-10-30 21:35:37.22978979 +0000 UTC m=+1537.359041340"
Oct 30 21:35:39.276331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:39.276300    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:35:41.034103 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:35:41.034001    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:35:41.623765 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.623731    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"]
Oct 30 21:35:41.628240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.628211    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.645827 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.645806    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-13658f-serving-cert\""
Oct 30 21:35:41.661510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.661479    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"]
Oct 30 21:35:41.763477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.763435    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.763664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.763503    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-metrics-certs\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.763664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.763650    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a0bd4-b176-4497-b22a-0b73cd039fcc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.763762 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.763674    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94k9h\" (UniqueName: \"kubernetes.io/projected/510a0bd4-b176-4497-b22a-0b73cd039fcc-kube-api-access-94k9h\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.864687 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.864657    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a0bd4-b176-4497-b22a-0b73cd039fcc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.864687 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.864690    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94k9h\" (UniqueName: \"kubernetes.io/projected/510a0bd4-b176-4497-b22a-0b73cd039fcc-kube-api-access-94k9h\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.864938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.864774    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.864938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.864819    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-metrics-certs\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.865483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.865451    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a0bd4-b176-4497-b22a-0b73cd039fcc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.867286 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.867262    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.867445 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.867417    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-metrics-certs\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.885302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.885238    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94k9h\" (UniqueName: \"kubernetes.io/projected/510a0bd4-b176-4497-b22a-0b73cd039fcc-kube-api-access-94k9h\") pod \"maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:41.942763 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:41.942729    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:42.263362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:42.263330    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"]
Oct 30 21:35:42.266098 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:35:42.266071    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510a0bd4_b176_4497_b22a_0b73cd039fcc.slice/crio-5bf38a4e02a426fe2ed435da83bada4c543d55dbb9708cf7ba3822786d1b7c6b WatchSource:0}: Error finding container 5bf38a4e02a426fe2ed435da83bada4c543d55dbb9708cf7ba3822786d1b7c6b: Status 404 returned error can't find the container with id 5bf38a4e02a426fe2ed435da83bada4c543d55dbb9708cf7ba3822786d1b7c6b
Oct 30 21:35:42.347574 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:42.347532    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj" event={"ID":"510a0bd4-b176-4497-b22a-0b73cd039fcc","Type":"ContainerStarted","Data":"5bf38a4e02a426fe2ed435da83bada4c543d55dbb9708cf7ba3822786d1b7c6b"}
Oct 30 21:35:43.343873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.343838    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"]
Oct 30 21:35:43.344267 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.344082    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn" podUID="60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" containerName="maas-api" containerID="cri-o://3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734" gracePeriod=30
Oct 30 21:35:43.357210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.357148    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj" event={"ID":"510a0bd4-b176-4497-b22a-0b73cd039fcc","Type":"ContainerStarted","Data":"cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a"}
Oct 30 21:35:43.357356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.357273    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:43.462096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.462026    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj" podStartSLOduration=2.108540575 podStartE2EDuration="2.462010567s" totalImagesPullingTime="353.469992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:43.458509455 +0000 UTC m=+1543.587760923" watchObservedRunningTime="2026-10-30 21:35:43.462010567 +0000 UTC m=+1543.591262024"
Oct 30 21:35:43.652287 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.652261    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:35:43.687483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.687447    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-metrics-certs\") pod \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") "
Oct 30 21:35:43.687674 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.687622    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-kube-api-access-8c26q\" (UniqueName: \"kubernetes.io/projected/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-kube-api-access-8c26q\") pod \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") "
Oct 30 21:35:43.687674 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.687665    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-openshift-service-ca\") pod \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") "
Oct 30 21:35:43.687792 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.687722    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-maas-api-tls\") pod \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\" (UID: \"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126\") "
Oct 30 21:35:43.688672 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.688640    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-openshift-service-ca" pod "60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" (UID: "60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:43.694300 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.694258    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-maas-api-tls" pod "60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" (UID: "60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:43.694417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.694295    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-kube-api-access-8c26q" pod "60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" (UID: "60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126"). InnerVolumeSpecName "kube-api-access-8c26q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:43.694417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.694298    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-metrics-certs" pod "60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" (UID: "60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:43.789056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.789012    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c26q\" (UniqueName: \"kubernetes.io/projected/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-kube-api-access-8c26q\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:43.789056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.789046    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:43.789056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.789060    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:43.789323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:43.789072    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:35:44.211006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.210969    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"]
Oct 30 21:35:44.211614 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.211597    5886 cpu_manager.go:414] "containerMap: removing container" podUID="60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" containerName="maas-api"
Oct 30 21:35:44.211614 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.211613    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" containerName="maas-api"
Oct 30 21:35:44.211824 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.211715    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" containerName="maas-api"
Oct 30 21:35:44.215405 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.215382    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.240787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.240754    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"]
Oct 30 21:35:44.241058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.241021    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-26a13f-serving-cert\""
Oct 30 21:35:44.294776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.294683    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.294931 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.294886    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6943443c-50b8-4113-8c1b-88ef3b2f057c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.295026 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.294995    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8s8j\" (UniqueName: \"kubernetes.io/projected/6943443c-50b8-4113-8c1b-88ef3b2f057c-kube-api-access-j8s8j\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.295146 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.295107    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-metrics-certs\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.365119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.365086    5886 generic.go:356] "Generic (PLEG): container finished" podID="60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" containerID="3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734" exitCode=0
Oct 30 21:35:44.365640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.365145    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"
Oct 30 21:35:44.365640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.365176    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn" event={"ID":"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126","Type":"ContainerDied","Data":"3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734"}
Oct 30 21:35:44.365640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.365327    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn" event={"ID":"60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126","Type":"ContainerDied","Data":"16036c954921d7da928cb71fec2bac043d49cacd62f3a0cc66e307ed53594a06"}
Oct 30 21:35:44.365640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.365360    5886 scope.go:122] "RemoveContainer" containerID="3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734"
Oct 30 21:35:44.376963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.376940    5886 scope.go:122] "RemoveContainer" containerID="3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734"
Oct 30 21:35:44.377295 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:35:44.377275    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734\": container with ID starting with 3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734 not found: ID does not exist" containerID="3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734"
Oct 30 21:35:44.377372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.377303    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734"} err="failed to get container status \"3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734\": rpc error: code = NotFound desc = could not find container \"3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734\": container with ID starting with 3b638073d167d3cfe4b35e7d51d24385c751a85d602817370f0c293b7ae50734 not found: ID does not exist"
Oct 30 21:35:44.396718 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.396694    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6943443c-50b8-4113-8c1b-88ef3b2f057c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.396843 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.396766    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8s8j\" (UniqueName: \"kubernetes.io/projected/6943443c-50b8-4113-8c1b-88ef3b2f057c-kube-api-access-j8s8j\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.396843 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.396813    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-metrics-certs\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.396970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.396950    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.397502 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.397451    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6943443c-50b8-4113-8c1b-88ef3b2f057c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.399392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.399371    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-metrics-certs\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.399612 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.399590    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.419303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.419273    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8s8j\" (UniqueName: \"kubernetes.io/projected/6943443c-50b8-4113-8c1b-88ef3b2f057c-kube-api-access-j8s8j\") pod \"maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.434521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.434500    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"]
Oct 30 21:35:44.443053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.443032    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3afc5e-779595c747-gtvgn"]
Oct 30 21:35:44.535208 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.534940    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:44.854486 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:44.854281    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"]
Oct 30 21:35:44.857467 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:35:44.857443    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6943443c_50b8_4113_8c1b_88ef3b2f057c.slice/crio-aaf2d26bda706021f821cffed592e37bf63c0d5ac2d0ba2effece59fb9ec7034 WatchSource:0}: Error finding container aaf2d26bda706021f821cffed592e37bf63c0d5ac2d0ba2effece59fb9ec7034: Status 404 returned error can't find the container with id aaf2d26bda706021f821cffed592e37bf63c0d5ac2d0ba2effece59fb9ec7034
Oct 30 21:35:45.375280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:45.375213    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf" event={"ID":"6943443c-50b8-4113-8c1b-88ef3b2f057c","Type":"ContainerStarted","Data":"61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c"}
Oct 30 21:35:45.375280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:45.375281    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf" event={"ID":"6943443c-50b8-4113-8c1b-88ef3b2f057c","Type":"ContainerStarted","Data":"aaf2d26bda706021f821cffed592e37bf63c0d5ac2d0ba2effece59fb9ec7034"}
Oct 30 21:35:45.375815 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:45.375309    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:35:45.431248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:45.431201    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf" podStartSLOduration=1.08893747 podStartE2EDuration="1.431187146s" totalImagesPullingTime="342.249676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:45.430899779 +0000 UTC m=+1545.560151240" watchObservedRunningTime="2026-10-30 21:35:45.431187146 +0000 UTC m=+1545.560438600"
Oct 30 21:35:45.996572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:45.996530    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126" path="/var/lib/kubelet/pods/60e4e2c3-1cc7-4caf-bbfa-bf4f365b4126/volumes"
Oct 30 21:35:49.372231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:49.372202    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:35:51.394612 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:35:51.394583    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:36:00.491628 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:00.491580    5886 generic.go:356] "Generic (PLEG): container finished" podID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerID="99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e" exitCode=0
Oct 30 21:36:00.492145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:00.491649    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" event={"ID":"f16f82a5-11e5-4c70-996f-a7d5ea36573a","Type":"ContainerDied","Data":"99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e"}
Oct 30 21:36:00.492145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:00.492128    5886 scope.go:122] "RemoveContainer" containerID="99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e"
Oct 30 21:36:00.493962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:00.493927    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:00.494066 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:00.494015    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:00.494119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:00.494061    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:36:01.501562 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:01.501526    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" event={"ID":"f16f82a5-11e5-4c70-996f-a7d5ea36573a","Type":"ContainerStarted","Data":"ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef"}
Oct 30 21:36:01.606036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:01.606002    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:36:01.606232 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:01.606212    5886 patch_prober.go:29] interesting pod/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:36:01.606300 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:01.606258    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 21:36:02.633178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:02.632554    5886 patch_prober.go:29] interesting pod/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:36:02.633178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:02.632615    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 21:36:03.606753 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:03.606708    5886 patch_prober.go:29] interesting pod/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:36:03.606938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:03.606769    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 21:36:04.606713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:04.606672    5886 patch_prober.go:29] interesting pod/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:36:04.607281 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:04.606770    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 21:36:05.606330 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:05.606286    5886 patch_prober.go:29] interesting pod/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:36:05.606544 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:05.606354    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 21:36:06.606642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:06.606597    5886 patch_prober.go:29] interesting pod/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:36:06.607148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:06.606661    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 21:36:07.610952 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:07.610918    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:36:07.611418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:07.611395    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:36:07.612141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:07.612117    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:36:11.029258 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:36:11.029223    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:36:27.255192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:27.255144    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"]
Oct 30 21:36:27.255770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:27.255419    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy" containerID="cri-o://5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e" gracePeriod=30
Oct 30 21:36:32.530646 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.530580    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:36:32.575549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575519    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-certs\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.575729 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575566    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-credential-socket\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.575729 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575625    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-envoy\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.575729 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575656    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-kube-api-access-9g2s6\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-kube-api-access-9g2s6\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.575729 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575687    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-token\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-token\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.576010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575741    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-data\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.576010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575768    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-podinfo\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.576010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575781    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-certs" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.576010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575835    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istiod-ca-cert\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.576010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575913    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-socket\") pod \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\" (UID: \"370db4fb-48b4-4d14-9200-b2d84e12c9f0\") "
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.575912    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-credential-socket" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.576083    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-data" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.576307    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-socket" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.576396    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istiod-ca-cert" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.576598    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.576621    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.576636    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.576651    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.576686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.576665    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.578239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.578210    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-envoy" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:36:32.578413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.578231    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-podinfo" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:36:32.578505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.578469    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-token" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:32.578832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.578809    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-kube-api-access-9g2s6" pod "370db4fb-48b4-4d14-9200-b2d84e12c9f0" (UID: "370db4fb-48b4-4d14-9200-b2d84e12c9f0"). InnerVolumeSpecName "kube-api-access-9g2s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:32.678494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.678457    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.678494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.678490    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g2s6\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-kube-api-access-9g2s6\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.678494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.678500    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.678735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.678508    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370db4fb-48b4-4d14-9200-b2d84e12c9f0-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:32.745824 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.745787    5886 generic.go:356] "Generic (PLEG): container finished" podID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerID="5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e" exitCode=0
Oct 30 21:36:32.746004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.745839    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"
Oct 30 21:36:32.746004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.745877    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" event={"ID":"370db4fb-48b4-4d14-9200-b2d84e12c9f0","Type":"ContainerDied","Data":"5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e"}
Oct 30 21:36:32.746004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.745937    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95" event={"ID":"370db4fb-48b4-4d14-9200-b2d84e12c9f0","Type":"ContainerDied","Data":"49734d8aa377e8ab0559c14ccca1352890dff172d1284fb1493ac6a35e36c625"}
Oct 30 21:36:32.746004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.745970    5886 scope.go:122] "RemoveContainer" containerID="5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e"
Oct 30 21:36:32.757172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.757135    5886 scope.go:122] "RemoveContainer" containerID="5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e"
Oct 30 21:36:32.757449 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:36:32.757424    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e\": container with ID starting with 5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e not found: ID does not exist" containerID="5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e"
Oct 30 21:36:32.757496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.757461    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e"} err="failed to get container status \"5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e\": rpc error: code = NotFound desc = could not find container \"5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e\": container with ID starting with 5fb90cd8dde8c58ac3f53763a20773df4199051b61962c4e2e4b6d5811bc5a9e not found: ID does not exist"
Oct 30 21:36:32.807088 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.807012    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"]
Oct 30 21:36:32.815714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:32.815685    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-6qx95"]
Oct 30 21:36:33.936353 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:33.936319    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"]
Oct 30 21:36:33.936743 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:33.936621    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg" podUID="b363130d-98c5-43ca-8dbf-40c05a5bc963" containerName="maas-api" containerID="cri-o://fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3" gracePeriod=30
Oct 30 21:36:33.989233 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:33.989198    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" path="/var/lib/kubelet/pods/370db4fb-48b4-4d14-9200-b2d84e12c9f0/volumes"
Oct 30 21:36:34.232573 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.232549    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:36:34.298529 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.298497    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b363130d-98c5-43ca-8dbf-40c05a5bc963-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b363130d-98c5-43ca-8dbf-40c05a5bc963-openshift-service-ca\") pod \"b363130d-98c5-43ca-8dbf-40c05a5bc963\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") "
Oct 30 21:36:34.298735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.298560    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-maas-api-tls\") pod \"b363130d-98c5-43ca-8dbf-40c05a5bc963\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") "
Oct 30 21:36:34.298735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.298588    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-metrics-certs\") pod \"b363130d-98c5-43ca-8dbf-40c05a5bc963\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") "
Oct 30 21:36:34.298735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.298649    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b363130d-98c5-43ca-8dbf-40c05a5bc963-kube-api-access-sqkfk\" (UniqueName: \"kubernetes.io/projected/b363130d-98c5-43ca-8dbf-40c05a5bc963-kube-api-access-sqkfk\") pod \"b363130d-98c5-43ca-8dbf-40c05a5bc963\" (UID: \"b363130d-98c5-43ca-8dbf-40c05a5bc963\") "
Oct 30 21:36:34.299301 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.299270    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b363130d-98c5-43ca-8dbf-40c05a5bc963-openshift-service-ca" pod "b363130d-98c5-43ca-8dbf-40c05a5bc963" (UID: "b363130d-98c5-43ca-8dbf-40c05a5bc963"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:34.314341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.308473    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b363130d-98c5-43ca-8dbf-40c05a5bc963-kube-api-access-sqkfk" pod "b363130d-98c5-43ca-8dbf-40c05a5bc963" (UID: "b363130d-98c5-43ca-8dbf-40c05a5bc963"). InnerVolumeSpecName "kube-api-access-sqkfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:34.314341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.308806    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-maas-api-tls" pod "b363130d-98c5-43ca-8dbf-40c05a5bc963" (UID: "b363130d-98c5-43ca-8dbf-40c05a5bc963"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:36:34.314341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.311700    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-metrics-certs" pod "b363130d-98c5-43ca-8dbf-40c05a5bc963" (UID: "b363130d-98c5-43ca-8dbf-40c05a5bc963"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:36:34.400858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.400818    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqkfk\" (UniqueName: \"kubernetes.io/projected/b363130d-98c5-43ca-8dbf-40c05a5bc963-kube-api-access-sqkfk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:34.400858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.400857    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b363130d-98c5-43ca-8dbf-40c05a5bc963-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:34.401082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.400872    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:34.401082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.400886    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b363130d-98c5-43ca-8dbf-40c05a5bc963-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:36:34.774000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.773967    5886 generic.go:356] "Generic (PLEG): container finished" podID="b363130d-98c5-43ca-8dbf-40c05a5bc963" containerID="fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3" exitCode=0
Oct 30 21:36:34.774246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.774034    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"
Oct 30 21:36:34.774246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.774050    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg" event={"ID":"b363130d-98c5-43ca-8dbf-40c05a5bc963","Type":"ContainerDied","Data":"fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3"}
Oct 30 21:36:34.774246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.774128    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg" event={"ID":"b363130d-98c5-43ca-8dbf-40c05a5bc963","Type":"ContainerDied","Data":"2c11d1babeccaa391ebcfd60b28589e718f08aaf65854f6c4995315790e47542"}
Oct 30 21:36:34.774246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.774175    5886 scope.go:122] "RemoveContainer" containerID="fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3"
Oct 30 21:36:34.785372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.785350    5886 scope.go:122] "RemoveContainer" containerID="fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3"
Oct 30 21:36:34.785970 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:36:34.785905    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3\": container with ID starting with fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3 not found: ID does not exist" containerID="fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3"
Oct 30 21:36:34.785970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.785948    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3"} err="failed to get container status \"fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3\": rpc error: code = NotFound desc = could not find container \"fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3\": container with ID starting with fb8cc8f85451a7274efd6560e35cae5556c3963ca63b44e557b4f149a7fdadb3 not found: ID does not exist"
Oct 30 21:36:34.846825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.846791    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"]
Oct 30 21:36:34.859315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.859286    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-38e5bf-6b77788988-jm2kg"]
Oct 30 21:36:34.885550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.885497    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"]
Oct 30 21:36:34.886248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.886227    5886 cpu_manager.go:414] "containerMap: removing container" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy"
Oct 30 21:36:34.886248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.886247    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy"
Oct 30 21:36:34.886419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.886301    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b363130d-98c5-43ca-8dbf-40c05a5bc963" containerName="maas-api"
Oct 30 21:36:34.886419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.886307    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b363130d-98c5-43ca-8dbf-40c05a5bc963" containerName="maas-api"
Oct 30 21:36:34.886419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.886401    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="370db4fb-48b4-4d14-9200-b2d84e12c9f0" containerName="istio-proxy"
Oct 30 21:36:34.886419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.886417    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b363130d-98c5-43ca-8dbf-40c05a5bc963" containerName="maas-api"
Oct 30 21:36:34.890856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.890834    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:34.917919 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.917896    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f05cda-serving-cert\""
Oct 30 21:36:34.935962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:34.935929    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"]
Oct 30 21:36:35.010988 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.010944    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-metrics-certs\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.011453 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.011058    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.011453 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.011282    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ed146a-6b47-46f5-8e7d-fafe61761bd1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.011453 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.011411    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qppw\" (UniqueName: \"kubernetes.io/projected/32ed146a-6b47-46f5-8e7d-fafe61761bd1-kube-api-access-4qppw\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.112833 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.112729    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ed146a-6b47-46f5-8e7d-fafe61761bd1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.113187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.113124    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qppw\" (UniqueName: \"kubernetes.io/projected/32ed146a-6b47-46f5-8e7d-fafe61761bd1-kube-api-access-4qppw\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.113320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.113293    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-metrics-certs\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.113384 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.113359    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.113735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.113702    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ed146a-6b47-46f5-8e7d-fafe61761bd1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.115997 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.115974    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-metrics-certs\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.116134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.116111    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.134516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.134487    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qppw\" (UniqueName: \"kubernetes.io/projected/32ed146a-6b47-46f5-8e7d-fafe61761bd1-kube-api-access-4qppw\") pod \"maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.208519 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.208479    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:35.530955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.530914    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"]
Oct 30 21:36:35.534388 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:36:35.534360    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ed146a_6b47_46f5_8e7d_fafe61761bd1.slice/crio-c1ae3a0bafdb3f84b63b582d50d31da36d184f6ce7669a566804c71ff8336fd9 WatchSource:0}: Error finding container c1ae3a0bafdb3f84b63b582d50d31da36d184f6ce7669a566804c71ff8336fd9: Status 404 returned error can't find the container with id c1ae3a0bafdb3f84b63b582d50d31da36d184f6ce7669a566804c71ff8336fd9
Oct 30 21:36:35.783977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.783889    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c" event={"ID":"32ed146a-6b47-46f5-8e7d-fafe61761bd1","Type":"ContainerStarted","Data":"c1ae3a0bafdb3f84b63b582d50d31da36d184f6ce7669a566804c71ff8336fd9"}
Oct 30 21:36:35.989642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:35.989606    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b363130d-98c5-43ca-8dbf-40c05a5bc963" path="/var/lib/kubelet/pods/b363130d-98c5-43ca-8dbf-40c05a5bc963/volumes"
Oct 30 21:36:36.797176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:36.797123    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c" event={"ID":"32ed146a-6b47-46f5-8e7d-fafe61761bd1","Type":"ContainerStarted","Data":"6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f"}
Oct 30 21:36:36.797739 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:36.797219    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:36.846362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:36.846311    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c" podStartSLOduration=2.335506306 podStartE2EDuration="2.846293574s" totalImagesPullingTime="510.787268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:36.843794762 +0000 UTC m=+1596.973046203" watchObservedRunningTime="2026-10-30 21:36:36.846293574 +0000 UTC m=+1596.975545031"
Oct 30 21:36:41.029005 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:36:41.028954    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:36:42.812785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:42.812753    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:36:47.023888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.023853    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"]
Oct 30 21:36:47.028366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.028347    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.056742 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.056719    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ea45c0-serving-cert\""
Oct 30 21:36:47.073901 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.073868    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"]
Oct 30 21:36:47.167114 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.167077    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.167320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.167202    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.167506 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.167488    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7j9c\" (UniqueName: \"kubernetes.io/projected/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-kube-api-access-q7j9c\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.167562 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.167523    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.269012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.268968    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7j9c\" (UniqueName: \"kubernetes.io/projected/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-kube-api-access-q7j9c\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.269228 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.269035    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.269228 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.269169    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.269367 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.269265    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.270854 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.270828    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.306246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.296065    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.309058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.309027    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-metrics-certs\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.316838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.316803    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7j9c\" (UniqueName: \"kubernetes.io/projected/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-kube-api-access-q7j9c\") pod \"maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.344363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.342742    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:47.733401 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.733367    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"]
Oct 30 21:36:47.737148 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:36:47.737115    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34928eaa_d2c3_4bd4_8f90_2553a7e538aa.slice/crio-5924b8eafd4dc40c60ef3ed87b202dd68c515087cd9bb6af01b07ca475e0e630 WatchSource:0}: Error finding container 5924b8eafd4dc40c60ef3ed87b202dd68c515087cd9bb6af01b07ca475e0e630: Status 404 returned error can't find the container with id 5924b8eafd4dc40c60ef3ed87b202dd68c515087cd9bb6af01b07ca475e0e630
Oct 30 21:36:47.880238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:47.880199    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8" event={"ID":"34928eaa-d2c3-4bd4-8f90-2553a7e538aa","Type":"ContainerStarted","Data":"5924b8eafd4dc40c60ef3ed87b202dd68c515087cd9bb6af01b07ca475e0e630"}
Oct 30 21:36:48.889606 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:48.889562    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8" event={"ID":"34928eaa-d2c3-4bd4-8f90-2553a7e538aa","Type":"ContainerStarted","Data":"39d99e22afb1bea2d08e88051457a96944fc661fdf12562e17b2d2df28c7c4aa"}
Oct 30 21:36:48.890064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:48.889809    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:48.935026 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:48.934965    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8" podStartSLOduration=2.59015169 podStartE2EDuration="2.934950027s" totalImagesPullingTime="344.798337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:48.931592483 +0000 UTC m=+1609.060843940" watchObservedRunningTime="2026-10-30 21:36:48.934950027 +0000 UTC m=+1609.064201484"
Oct 30 21:36:53.917009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:53.916967    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"]
Oct 30 21:36:53.921540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:53.921516    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:53.941138 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:53.941108    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-8a20c6-serving-cert\""
Oct 30 21:36:53.956262 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:53.956230    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec32f4e7-1385-49af-af57-653322f0d878-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:53.956473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:53.956341    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"]
Oct 30 21:36:53.956473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:53.956468    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fmg\" (UniqueName: \"kubernetes.io/projected/ec32f4e7-1385-49af-af57-653322f0d878-kube-api-access-g4fmg\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:53.956962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:53.956572    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:53.956962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:53.956608    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-metrics-certs\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.057526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.057489    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fmg\" (UniqueName: \"kubernetes.io/projected/ec32f4e7-1385-49af-af57-653322f0d878-kube-api-access-g4fmg\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.057714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.057577    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.057714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.057603    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-metrics-certs\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.057910 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.057888    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec32f4e7-1385-49af-af57-653322f0d878-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.058701 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.058675    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec32f4e7-1385-49af-af57-653322f0d878-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.060514 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.060487    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-metrics-certs\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.060630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.060489    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.077125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.077094    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fmg\" (UniqueName: \"kubernetes.io/projected/ec32f4e7-1385-49af-af57-653322f0d878-kube-api-access-g4fmg\") pod \"maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.236474 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.236382    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:54.547936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.547854    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"]
Oct 30 21:36:54.551229 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:36:54.551197    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec32f4e7_1385_49af_af57_653322f0d878.slice/crio-4944b588fe5fa8f2aceef9a4220b935e2e929cf03c88c5b7cdfa22ef54397cd5 WatchSource:0}: Error finding container 4944b588fe5fa8f2aceef9a4220b935e2e929cf03c88c5b7cdfa22ef54397cd5: Status 404 returned error can't find the container with id 4944b588fe5fa8f2aceef9a4220b935e2e929cf03c88c5b7cdfa22ef54397cd5
Oct 30 21:36:54.905076 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.905054    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:36:54.949493 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:54.949456    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc" event={"ID":"ec32f4e7-1385-49af-af57-653322f0d878","Type":"ContainerStarted","Data":"4944b588fe5fa8f2aceef9a4220b935e2e929cf03c88c5b7cdfa22ef54397cd5"}
Oct 30 21:36:55.958366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:55.958330    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc" event={"ID":"ec32f4e7-1385-49af-af57-653322f0d878","Type":"ContainerStarted","Data":"cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1"}
Oct 30 21:36:55.958763 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:55.958519    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:36:56.002532 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:36:56.002480    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc" podStartSLOduration=2.652338248 podStartE2EDuration="3.002466351s" totalImagesPullingTime="350.128103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:56.001044947 +0000 UTC m=+1616.130296430" watchObservedRunningTime="2026-10-30 21:36:56.002466351 +0000 UTC m=+1616.131717807"
Oct 30 21:37:01.284687 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:01.284656    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-6l9b4_ee881cfc-ad3c-4017-8504-722b0543c67f/payload-processing/0.log"
Oct 30 21:37:01.566251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:01.566127    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-6l9b4_ee881cfc-ad3c-4017-8504-722b0543c67f/payload-processing/0.log"
Oct 30 21:37:01.973187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:01.973136    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:37:11.030672 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:11.030585    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:37:16.309633 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.309596    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"]
Oct 30 21:37:16.315634 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.315616    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.333269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.333240    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-912e61-serving-cert\""
Oct 30 21:37:16.338867 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.338833    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"]
Oct 30 21:37:16.382442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.382406    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-metrics-certs\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.382634 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.382551    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.382758 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.382734    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e514d66-470e-4220-98fb-71057b79cf0a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.382895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.382863    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4lx\" (UniqueName: \"kubernetes.io/projected/6e514d66-470e-4220-98fb-71057b79cf0a-kube-api-access-6t4lx\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.484630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.484583    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4lx\" (UniqueName: \"kubernetes.io/projected/6e514d66-470e-4220-98fb-71057b79cf0a-kube-api-access-6t4lx\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.484836 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.484679    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-metrics-certs\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.484836 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.484763    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.485116 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.485097    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e514d66-470e-4220-98fb-71057b79cf0a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.485903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.485873    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e514d66-470e-4220-98fb-71057b79cf0a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.487352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.487328    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-metrics-certs\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.487439 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.487392    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.503042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.503011    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4lx\" (UniqueName: \"kubernetes.io/projected/6e514d66-470e-4220-98fb-71057b79cf0a-kube-api-access-6t4lx\") pod \"maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.632483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.632388    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:16.947787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:16.947753    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"]
Oct 30 21:37:16.951285 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:37:16.951255    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e514d66_470e_4220_98fb_71057b79cf0a.slice/crio-31aafa65fe54c1fdf43f6f41cd1dee3e5ed1ad50f96735de71a10c8137be141e WatchSource:0}: Error finding container 31aafa65fe54c1fdf43f6f41cd1dee3e5ed1ad50f96735de71a10c8137be141e: Status 404 returned error can't find the container with id 31aafa65fe54c1fdf43f6f41cd1dee3e5ed1ad50f96735de71a10c8137be141e
Oct 30 21:37:17.126975 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:17.126930    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2" event={"ID":"6e514d66-470e-4220-98fb-71057b79cf0a","Type":"ContainerStarted","Data":"31aafa65fe54c1fdf43f6f41cd1dee3e5ed1ad50f96735de71a10c8137be141e"}
Oct 30 21:37:18.135874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:18.135833    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2" event={"ID":"6e514d66-470e-4220-98fb-71057b79cf0a","Type":"ContainerStarted","Data":"07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0"}
Oct 30 21:37:18.136319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:18.135994    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:18.176385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:18.176333    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2" podStartSLOduration=1.889699195 podStartE2EDuration="2.176318947s" totalImagesPullingTime="286.619752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:18.172321968 +0000 UTC m=+1638.301573426" watchObservedRunningTime="2026-10-30 21:37:18.176318947 +0000 UTC m=+1638.305570403"
Oct 30 21:37:24.149467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.149436    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:37:24.248147 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.248115    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"]
Oct 30 21:37:24.248419 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.248376    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf" podUID="e91f8d27-558e-4456-89d5-daf1c1c1ebe6" containerName="maas-api" containerID="cri-o://5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952" gracePeriod=30
Oct 30 21:37:24.277831 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.277593    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf" podUID="e91f8d27-558e-4456-89d5-daf1c1c1ebe6" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.119:8443/health\": dial tcp 10.133.0.119:8443: connect: connection refused"
Oct 30 21:37:24.541882 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.541856    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:37:24.648458 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.648420    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-kube-api-access-jh5b4\" (UniqueName: \"kubernetes.io/projected/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-kube-api-access-jh5b4\") pod \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") "
Oct 30 21:37:24.648646 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.648524    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-metrics-certs\") pod \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") "
Oct 30 21:37:24.648646 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.648621    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-openshift-service-ca\") pod \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") "
Oct 30 21:37:24.648766 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.648739    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-maas-api-tls\") pod \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\" (UID: \"e91f8d27-558e-4456-89d5-daf1c1c1ebe6\") "
Oct 30 21:37:24.648984 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.648953    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-openshift-service-ca" pod "e91f8d27-558e-4456-89d5-daf1c1c1ebe6" (UID: "e91f8d27-558e-4456-89d5-daf1c1c1ebe6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:24.649306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.649282    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:24.650611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.650583    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-kube-api-access-jh5b4" pod "e91f8d27-558e-4456-89d5-daf1c1c1ebe6" (UID: "e91f8d27-558e-4456-89d5-daf1c1c1ebe6"). InnerVolumeSpecName "kube-api-access-jh5b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:24.650713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.650583    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-metrics-certs" pod "e91f8d27-558e-4456-89d5-daf1c1c1ebe6" (UID: "e91f8d27-558e-4456-89d5-daf1c1c1ebe6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:24.650713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.650669    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-maas-api-tls" pod "e91f8d27-558e-4456-89d5-daf1c1c1ebe6" (UID: "e91f8d27-558e-4456-89d5-daf1c1c1ebe6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:24.778323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.773634    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:24.778323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.773672    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5b4\" (UniqueName: \"kubernetes.io/projected/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-kube-api-access-jh5b4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:24.778323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:24.773685    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e91f8d27-558e-4456-89d5-daf1c1c1ebe6-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:25.191339 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.191303    5886 generic.go:356] "Generic (PLEG): container finished" podID="e91f8d27-558e-4456-89d5-daf1c1c1ebe6" containerID="5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952" exitCode=0
Oct 30 21:37:25.191834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.191348    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"
Oct 30 21:37:25.191834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.191380    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf" event={"ID":"e91f8d27-558e-4456-89d5-daf1c1c1ebe6","Type":"ContainerDied","Data":"5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952"}
Oct 30 21:37:25.191834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.191437    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf" event={"ID":"e91f8d27-558e-4456-89d5-daf1c1c1ebe6","Type":"ContainerDied","Data":"e6399bbf19e3f5538721a5f5dbd31467ba8033f37108c0e759113ae60130fe3e"}
Oct 30 21:37:25.191834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.191467    5886 scope.go:122] "RemoveContainer" containerID="5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952"
Oct 30 21:37:25.204857 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.204824    5886 scope.go:122] "RemoveContainer" containerID="5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952"
Oct 30 21:37:25.206733 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:25.206707    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952\": container with ID starting with 5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952 not found: ID does not exist" containerID="5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952"
Oct 30 21:37:25.206845 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.206741    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952"} err="failed to get container status \"5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952\": rpc error: code = NotFound desc = could not find container \"5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952\": container with ID starting with 5fb811695d8026646b05ca0afa87057dee00907e3e10b663f29eadfd28384952 not found: ID does not exist"
Oct 30 21:37:25.243553 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.243522    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"]
Oct 30 21:37:25.254587 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.254563    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3a5c92-788dd7f459-6mzzf"]
Oct 30 21:37:25.991788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:25.991751    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91f8d27-558e-4456-89d5-daf1c1c1ebe6" path="/var/lib/kubelet/pods/e91f8d27-558e-4456-89d5-daf1c1c1ebe6/volumes"
Oct 30 21:37:26.843398 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:26.843297    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"]
Oct 30 21:37:26.843845 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:26.843508    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy" containerID="cri-o://6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e" gracePeriod=30
Oct 30 21:37:30.106431 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:30.106396    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"]
Oct 30 21:37:30.106887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:30.106594    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy" containerID="cri-o://88af22afaac3bd04f6fb135addad52784145dbd1e91096f0b979b15d92c863b9" gracePeriod=30
Oct 30 21:37:30.117827 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:30.117798    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"]
Oct 30 21:37:30.118030 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:30.118005    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy" containerID="cri-o://27870abff2caf76f51d74c4320fae1d2c12e01b61338385b35f130cd31b8725b" gracePeriod=30
Oct 30 21:37:32.130629 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.130548    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:37:32.250711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.250674    5886 generic.go:356] "Generic (PLEG): container finished" podID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerID="6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e" exitCode=0
Oct 30 21:37:32.250907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.250725    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"
Oct 30 21:37:32.250907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.250755    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" event={"ID":"2930b56b-c0fe-46a3-a059-04de254fb9f8","Type":"ContainerDied","Data":"6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e"}
Oct 30 21:37:32.250907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.250811    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96" event={"ID":"2930b56b-c0fe-46a3-a059-04de254fb9f8","Type":"ContainerDied","Data":"87ca7aee26aa418ea6513bffab860bf9376a2d49d2b27f3246f76b53259e0240"}
Oct 30 21:37:32.250907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.250838    5886 scope.go:122] "RemoveContainer" containerID="6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e"
Oct 30 21:37:32.261128 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.261107    5886 scope.go:122] "RemoveContainer" containerID="e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9"
Oct 30 21:37:32.272999 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.272954    5886 scope.go:122] "RemoveContainer" containerID="6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e"
Oct 30 21:37:32.273314 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:32.273294    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e\": container with ID starting with 6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e not found: ID does not exist" containerID="6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e"
Oct 30 21:37:32.273407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.273330    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e"} err="failed to get container status \"6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e\": rpc error: code = NotFound desc = could not find container \"6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e\": container with ID starting with 6aef3282864f22b1e8738924c031f467b61febea284cbf1cb22d55a60580050e not found: ID does not exist"
Oct 30 21:37:32.273407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.273348    5886 scope.go:122] "RemoveContainer" containerID="e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9"
Oct 30 21:37:32.273666 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:32.273635    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9\": container with ID starting with e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9 not found: ID does not exist" containerID="e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9"
Oct 30 21:37:32.273734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.273671    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9"} err="failed to get container status \"e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9\": rpc error: code = NotFound desc = could not find container \"e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9\": container with ID starting with e2fc8f212663d3afbb2a873ff38aa7b805dafc4a28b68afd3761a4a9e27b31b9 not found: ID does not exist"
Oct 30 21:37:32.282113 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282080    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-envoy\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.282279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282236    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-credential-socket\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.282359 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282341    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-socket\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.282600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282574    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-credential-socket" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:32.282661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282601    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-socket" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:32.282661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282612    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-token\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-token\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.282746 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282692    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2930b56b-c0fe-46a3-a059-04de254fb9f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2930b56b-c0fe-46a3-a059-04de254fb9f8-istiod-ca-cert\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.282746 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282721    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-kube-api-access-7fx27\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-kube-api-access-7fx27\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.282856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282759    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-podinfo\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.282856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282823    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-data\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.282938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.282903    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-certs\") pod \"2930b56b-c0fe-46a3-a059-04de254fb9f8\" (UID: \"2930b56b-c0fe-46a3-a059-04de254fb9f8\") "
Oct 30 21:37:32.283599 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.283055    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2930b56b-c0fe-46a3-a059-04de254fb9f8-istiod-ca-cert" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:32.283727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.283627    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2930b56b-c0fe-46a3-a059-04de254fb9f8-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.283727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.283648    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.283727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.283663    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.284129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.283888    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-data" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:32.284129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.283969    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-certs" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:32.287201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.287168    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-token" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:32.287201 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.287183    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-kube-api-access-7fx27" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "kube-api-access-7fx27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:32.287444 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.287420    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-podinfo" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:32.291467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.291443    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-envoy" pod "2930b56b-c0fe-46a3-a059-04de254fb9f8" (UID: "2930b56b-c0fe-46a3-a059-04de254fb9f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:32.385438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.385351    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.385438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.385386    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fx27\" (UniqueName: \"kubernetes.io/projected/2930b56b-c0fe-46a3-a059-04de254fb9f8-kube-api-access-7fx27\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.385438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.385399    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.385438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.385410    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.385438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.385422    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.385438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.385434    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2930b56b-c0fe-46a3-a059-04de254fb9f8-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:32.598086 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.598057    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"]
Oct 30 21:37:32.609117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.609089    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-7qg96"]
Oct 30 21:37:32.711536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.711496    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"]
Oct 30 21:37:32.711806 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:32.711760    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy" containerID="cri-o://ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef" gracePeriod=30
Oct 30 21:37:33.989558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:33.989526    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" path="/var/lib/kubelet/pods/2930b56b-c0fe-46a3-a059-04de254fb9f8/volumes"
Oct 30 21:37:34.287824 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:34.287740    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"]
Oct 30 21:37:34.323257 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:34.323228    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"]
Oct 30 21:37:35.292773 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.292530    5886 generic.go:356] "Generic (PLEG): container finished" podID="c2824625-1259-44b3-beac-44087a9d94ad" containerID="88af22afaac3bd04f6fb135addad52784145dbd1e91096f0b979b15d92c863b9" exitCode=0
Oct 30 21:37:35.292773 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.292571    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" event={"ID":"c2824625-1259-44b3-beac-44087a9d94ad","Type":"ContainerDied","Data":"88af22afaac3bd04f6fb135addad52784145dbd1e91096f0b979b15d92c863b9"}
Oct 30 21:37:35.297752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.297611    5886 generic.go:356] "Generic (PLEG): container finished" podID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerID="27870abff2caf76f51d74c4320fae1d2c12e01b61338385b35f130cd31b8725b" exitCode=0
Oct 30 21:37:35.297752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.297695    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" event={"ID":"a233a4d2-3e10-4170-ac3f-d11fc6302225","Type":"ContainerDied","Data":"27870abff2caf76f51d74c4320fae1d2c12e01b61338385b35f130cd31b8725b"}
Oct 30 21:37:35.415776 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.415743    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:37:35.432724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.432689    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-credential-socket\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.432885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.432842    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-envoy\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.432885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.432873    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-socket\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.432989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.432915    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-certs\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.432989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.432940    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-credential-socket" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.432989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.432973    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2824625-1259-44b3-beac-44087a9d94ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2824625-1259-44b3-beac-44087a9d94ad-istio-podinfo\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.433136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433002    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-istio-token\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-istio-token\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.433136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433074    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2824625-1259-44b3-beac-44087a9d94ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2824625-1259-44b3-beac-44087a9d94ad-istiod-ca-cert\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.433136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433105    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-data\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.433293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433139    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-kube-api-access-kmn6l\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-kube-api-access-kmn6l\") pod \"c2824625-1259-44b3-beac-44087a9d94ad\" (UID: \"c2824625-1259-44b3-beac-44087a9d94ad\") "
Oct 30 21:37:35.433293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433172    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-socket" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.433293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433213    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-certs" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.433720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433696    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.433720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433721    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.433889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433736    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.433889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.433746    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2824625-1259-44b3-beac-44087a9d94ad-istiod-ca-cert" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:35.434074 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.434047    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-data" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.436020 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.435540    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-envoy" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.436137 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.436086    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-kube-api-access-kmn6l" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "kube-api-access-kmn6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:35.436365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.436346    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-istio-token" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:35.437145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.437120    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2824625-1259-44b3-beac-44087a9d94ad-istio-podinfo" pod "c2824625-1259-44b3-beac-44087a9d94ad" (UID: "c2824625-1259-44b3-beac-44087a9d94ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:35.460879 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.460859    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.534714    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-certs\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.534773    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a233a4d2-3e10-4170-ac3f-d11fc6302225-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a233a4d2-3e10-4170-ac3f-d11fc6302225-istiod-ca-cert\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.534806    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-socket\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.534866    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-data\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.534901    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-token\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-token\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.534951    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-envoy\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.535099    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-podinfo\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.535179    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-kube-api-access-lj88m\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-kube-api-access-lj88m\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.535228    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-credential-socket\") pod \"a233a4d2-3e10-4170-ac3f-d11fc6302225\" (UID: \"a233a4d2-3e10-4170-ac3f-d11fc6302225\") "
Oct 30 21:37:35.535962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.535481    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-certs" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.535962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.535691    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-credential-socket" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.535962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.535828    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-data" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.536238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.536199    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-socket" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.536238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.536207    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a233a4d2-3e10-4170-ac3f-d11fc6302225-istiod-ca-cert" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:35.538067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538038    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2824625-1259-44b3-beac-44087a9d94ad-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538068    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538068    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-envoy" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:35.538248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538082    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2824625-1259-44b3-beac-44087a9d94ad-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538096    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538110    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a233a4d2-3e10-4170-ac3f-d11fc6302225-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538122    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538135    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538544 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538149    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmn6l\" (UniqueName: \"kubernetes.io/projected/c2824625-1259-44b3-beac-44087a9d94ad-kube-api-access-kmn6l\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538544 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538292    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538544 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538307    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538544 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538319    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2824625-1259-44b3-beac-44087a9d94ad-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.538757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.538724    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-token" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:35.539052 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.539015    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-kube-api-access-lj88m" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "kube-api-access-lj88m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:35.539210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.539062    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-podinfo" pod "a233a4d2-3e10-4170-ac3f-d11fc6302225" (UID: "a233a4d2-3e10-4170-ac3f-d11fc6302225"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:35.639097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.639018    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.639097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.639046    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj88m\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-kube-api-access-lj88m\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.639097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.639056    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:35.639097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:35.639071    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a233a4d2-3e10-4170-ac3f-d11fc6302225-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:36.308248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.308127    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw" event={"ID":"c2824625-1259-44b3-beac-44087a9d94ad","Type":"ContainerDied","Data":"de764fa52bfa97b7a676192c2a86d94d269fc909a13e070e33e517ee351ce52e"}
Oct 30 21:37:36.308248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.308231    5886 scope.go:122] "RemoveContainer" containerID="88af22afaac3bd04f6fb135addad52784145dbd1e91096f0b979b15d92c863b9"
Oct 30 21:37:36.308761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.308140    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"
Oct 30 21:37:36.311673 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.311651    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"
Oct 30 21:37:36.311673 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.311659    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm" event={"ID":"a233a4d2-3e10-4170-ac3f-d11fc6302225","Type":"ContainerDied","Data":"4dd00863b1cf5c35f71a1f842f3b476d718acb5c32513c8a1062188e1c94da03"}
Oct 30 21:37:36.319784 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.319766    5886 scope.go:122] "RemoveContainer" containerID="27870abff2caf76f51d74c4320fae1d2c12e01b61338385b35f130cd31b8725b"
Oct 30 21:37:36.357394 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.357371    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"]
Oct 30 21:37:36.370028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.369999    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6jsjm"]
Oct 30 21:37:36.399549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.399506    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"]
Oct 30 21:37:36.403687 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:36.403661    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-6bhtw"]
Oct 30 21:37:37.614093 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:37.614052    5886 patch_prober.go:29] interesting pod/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:37:37.614486 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:37.614119    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:37:37.990547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:37.990513    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" path="/var/lib/kubelet/pods/a233a4d2-3e10-4170-ac3f-d11fc6302225/volumes"
Oct 30 21:37:37.991214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:37.991195    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2824625-1259-44b3-beac-44087a9d94ad" path="/var/lib/kubelet/pods/c2824625-1259-44b3-beac-44087a9d94ad/volumes"
Oct 30 21:37:38.007882 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.007861    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071065    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-socket\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071120    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-podinfo\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071263    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-token\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-token\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071292    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istiod-ca-cert\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071323    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-kube-api-access-88db8\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-kube-api-access-88db8\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071380    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-certs\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071479    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-data\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071507    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-credential-socket\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.071552    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-envoy\") pod \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\" (UID: \"f16f82a5-11e5-4c70-996f-a7d5ea36573a\") "
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.072202    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-socket" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.072220    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-data" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.072296    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istiod-ca-cert" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.072420    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-certs" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:38.072923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.072538    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-credential-socket" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:38.075055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.075023    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-token" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:38.076285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.076258    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-envoy" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:38.081513 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.081486    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-podinfo" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:38.083224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.083195    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-kube-api-access-88db8" pod "f16f82a5-11e5-4c70-996f-a7d5ea36573a" (UID: "f16f82a5-11e5-4c70-996f-a7d5ea36573a"). InnerVolumeSpecName "kube-api-access-88db8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:38.173602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173575    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.173602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173602    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.173781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173613    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.173781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173622    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.173781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173630    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.173781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173640    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.173781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173647    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f16f82a5-11e5-4c70-996f-a7d5ea36573a-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.173781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173655    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88db8\" (UniqueName: \"kubernetes.io/projected/f16f82a5-11e5-4c70-996f-a7d5ea36573a-kube-api-access-88db8\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.173781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.173663    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f16f82a5-11e5-4c70-996f-a7d5ea36573a-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:38.329624 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.329592    5886 generic.go:356] "Generic (PLEG): container finished" podID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerID="ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef" exitCode=0
Oct 30 21:37:38.329830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.329642    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"
Oct 30 21:37:38.329830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.329682    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" event={"ID":"f16f82a5-11e5-4c70-996f-a7d5ea36573a","Type":"ContainerDied","Data":"ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef"}
Oct 30 21:37:38.329830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.329745    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86" event={"ID":"f16f82a5-11e5-4c70-996f-a7d5ea36573a","Type":"ContainerDied","Data":"66d1af4820d80bb20215bebac8f2cfb629b0342a025821a0ff8dddaa9d34f508"}
Oct 30 21:37:38.329830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.329777    5886 scope.go:122] "RemoveContainer" containerID="ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef"
Oct 30 21:37:38.340372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.340356    5886 scope.go:122] "RemoveContainer" containerID="99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e"
Oct 30 21:37:38.349350 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.349334    5886 scope.go:122] "RemoveContainer" containerID="ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef"
Oct 30 21:37:38.349618 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:38.349596    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef\": container with ID starting with ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef not found: ID does not exist" containerID="ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef"
Oct 30 21:37:38.349690 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.349627    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef"} err="failed to get container status \"ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef\": rpc error: code = NotFound desc = could not find container \"ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef\": container with ID starting with ef2d311260f527438718bf8612edd490e2d0b4f0e81fda771ab965528488aaef not found: ID does not exist"
Oct 30 21:37:38.349690 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.349650    5886 scope.go:122] "RemoveContainer" containerID="99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e"
Oct 30 21:37:38.349920 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:38.349901    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e\": container with ID starting with 99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e not found: ID does not exist" containerID="99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e"
Oct 30 21:37:38.349972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.349927    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e"} err="failed to get container status \"99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e\": rpc error: code = NotFound desc = could not find container \"99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e\": container with ID starting with 99f639ea5e0c1142ba27693a2e015183b567be6ecd252b0af3ab4781ac23651e not found: ID does not exist"
Oct 30 21:37:38.383825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.383792    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"]
Oct 30 21:37:38.392828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.392803    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3a5c92-openshift-default-555fd8b5c7-svw86"]
Oct 30 21:37:38.815914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:38.815883    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"]
Oct 30 21:37:39.288979 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.288923    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm" podUID="421b5cbd-ab97-449d-8330-85d7b653982e" containerName="payload-processing" containerID="cri-o://8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715" gracePeriod=25
Oct 30 21:37:39.324242 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.324206    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg" podUID="b21188ce-77a0-4112-9773-6a1fa0a1f955" containerName="payload-pre-processing" containerID="cri-o://29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972" gracePeriod=25
Oct 30 21:37:39.644073 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.644032    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:37:39.661989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.661962    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:37:39.692639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.692615    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421b5cbd-ab97-449d-8330-85d7b653982e-kube-api-access-vfgmm\" (UniqueName: \"kubernetes.io/projected/421b5cbd-ab97-449d-8330-85d7b653982e-kube-api-access-vfgmm\") pod \"421b5cbd-ab97-449d-8330-85d7b653982e\" (UID: \"421b5cbd-ab97-449d-8330-85d7b653982e\") "
Oct 30 21:37:39.692774 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.692650    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21188ce-77a0-4112-9773-6a1fa0a1f955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21188ce-77a0-4112-9773-6a1fa0a1f955-plugins-config-volume\") pod \"b21188ce-77a0-4112-9773-6a1fa0a1f955\" (UID: \"b21188ce-77a0-4112-9773-6a1fa0a1f955\") "
Oct 30 21:37:39.692774 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.692731    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21188ce-77a0-4112-9773-6a1fa0a1f955-kube-api-access-jkxq5\" (UniqueName: \"kubernetes.io/projected/b21188ce-77a0-4112-9773-6a1fa0a1f955-kube-api-access-jkxq5\") pod \"b21188ce-77a0-4112-9773-6a1fa0a1f955\" (UID: \"b21188ce-77a0-4112-9773-6a1fa0a1f955\") "
Oct 30 21:37:39.692929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.692896    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421b5cbd-ab97-449d-8330-85d7b653982e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421b5cbd-ab97-449d-8330-85d7b653982e-plugins-config-volume\") pod \"421b5cbd-ab97-449d-8330-85d7b653982e\" (UID: \"421b5cbd-ab97-449d-8330-85d7b653982e\") "
Oct 30 21:37:39.693127 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.693103    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21188ce-77a0-4112-9773-6a1fa0a1f955-plugins-config-volume" pod "b21188ce-77a0-4112-9773-6a1fa0a1f955" (UID: "b21188ce-77a0-4112-9773-6a1fa0a1f955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:39.693449 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.693425    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b21188ce-77a0-4112-9773-6a1fa0a1f955-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.693578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.693483    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421b5cbd-ab97-449d-8330-85d7b653982e-plugins-config-volume" pod "421b5cbd-ab97-449d-8330-85d7b653982e" (UID: "421b5cbd-ab97-449d-8330-85d7b653982e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:39.695306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.695284    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421b5cbd-ab97-449d-8330-85d7b653982e-kube-api-access-vfgmm" pod "421b5cbd-ab97-449d-8330-85d7b653982e" (UID: "421b5cbd-ab97-449d-8330-85d7b653982e"). InnerVolumeSpecName "kube-api-access-vfgmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:39.695432 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.695405    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21188ce-77a0-4112-9773-6a1fa0a1f955-kube-api-access-jkxq5" pod "b21188ce-77a0-4112-9773-6a1fa0a1f955" (UID: "b21188ce-77a0-4112-9773-6a1fa0a1f955"). InnerVolumeSpecName "kube-api-access-jkxq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:39.794843 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.794809    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxq5\" (UniqueName: \"kubernetes.io/projected/b21188ce-77a0-4112-9773-6a1fa0a1f955-kube-api-access-jkxq5\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.794843 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.794846    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421b5cbd-ab97-449d-8330-85d7b653982e-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.795054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.794863    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgmm\" (UniqueName: \"kubernetes.io/projected/421b5cbd-ab97-449d-8330-85d7b653982e-kube-api-access-vfgmm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:39.992387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:39.992353    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" path="/var/lib/kubelet/pods/f16f82a5-11e5-4c70-996f-a7d5ea36573a/volumes"
Oct 30 21:37:40.345658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.345628    5886 generic.go:356] "Generic (PLEG): container finished" podID="b21188ce-77a0-4112-9773-6a1fa0a1f955" containerID="29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972" exitCode=0
Oct 30 21:37:40.345799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.345673    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"
Oct 30 21:37:40.345799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.345705    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg" event={"ID":"b21188ce-77a0-4112-9773-6a1fa0a1f955","Type":"ContainerDied","Data":"29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972"}
Oct 30 21:37:40.345799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.345755    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg" event={"ID":"b21188ce-77a0-4112-9773-6a1fa0a1f955","Type":"ContainerDied","Data":"bf87fb1d8512f03066d86a605da30b8981bbeacd979208bd6e25bf4474f3040e"}
Oct 30 21:37:40.345799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.345781    5886 scope.go:122] "RemoveContainer" containerID="29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972"
Oct 30 21:37:40.347290 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.347269    5886 generic.go:356] "Generic (PLEG): container finished" podID="421b5cbd-ab97-449d-8330-85d7b653982e" containerID="8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715" exitCode=0
Oct 30 21:37:40.347396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.347382    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm" event={"ID":"421b5cbd-ab97-449d-8330-85d7b653982e","Type":"ContainerDied","Data":"8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715"}
Oct 30 21:37:40.347447 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.347423    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm" event={"ID":"421b5cbd-ab97-449d-8330-85d7b653982e","Type":"ContainerDied","Data":"699fc7016d906904e9ffef3ad325e9316b99105001907b91b0e3c67f55279919"}
Oct 30 21:37:40.347504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.347473    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"
Oct 30 21:37:40.355335 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.355318    5886 scope.go:122] "RemoveContainer" containerID="29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972"
Oct 30 21:37:40.355591 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:40.355575    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972\": container with ID starting with 29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972 not found: ID does not exist" containerID="29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972"
Oct 30 21:37:40.355642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.355599    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972"} err="failed to get container status \"29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972\": rpc error: code = NotFound desc = could not find container \"29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972\": container with ID starting with 29305ca31be289fb2fd2b5fb1b6973624f60d07b949b338b0ba2ca02df939972 not found: ID does not exist"
Oct 30 21:37:40.355642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.355613    5886 scope.go:122] "RemoveContainer" containerID="8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715"
Oct 30 21:37:40.365185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.365145    5886 scope.go:122] "RemoveContainer" containerID="8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715"
Oct 30 21:37:40.365470 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:40.365446    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715\": container with ID starting with 8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715 not found: ID does not exist" containerID="8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715"
Oct 30 21:37:40.365539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.365480    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715"} err="failed to get container status \"8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715\": rpc error: code = NotFound desc = could not find container \"8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715\": container with ID starting with 8a1159b9b87f08208d64bda34baae4472b23dd397ac9ce76f6e29d5625354715 not found: ID does not exist"
Oct 30 21:37:40.394628 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.394600    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"]
Oct 30 21:37:40.406139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.406100    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-13658f-655697579f-p6jbg"]
Oct 30 21:37:40.431883 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.431859    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"]
Oct 30 21:37:40.437028 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:40.437006    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-13658f-5dbb78bc4f-pzhnm"]
Oct 30 21:37:41.034025 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:41.033981    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:37:41.990018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:41.989977    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421b5cbd-ab97-449d-8330-85d7b653982e" path="/var/lib/kubelet/pods/421b5cbd-ab97-449d-8330-85d7b653982e/volumes"
Oct 30 21:37:41.990521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:41.990498    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21188ce-77a0-4112-9773-6a1fa0a1f955" path="/var/lib/kubelet/pods/b21188ce-77a0-4112-9773-6a1fa0a1f955/volumes"
Oct 30 21:37:43.816844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:43.816797    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x" podUID="b55af1b4-7a9b-4a6f-8801-481e67895f4a" containerName="payload-processing" containerID="cri-o://39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1" gracePeriod=25
Oct 30 21:37:44.106598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.106564    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:37:44.149454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.149418    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b55af1b4-7a9b-4a6f-8801-481e67895f4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b55af1b4-7a9b-4a6f-8801-481e67895f4a-plugins-config-volume\") pod \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\" (UID: \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\") "
Oct 30 21:37:44.149621 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.149499    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55af1b4-7a9b-4a6f-8801-481e67895f4a-kube-api-access-lqkvk\" (UniqueName: \"kubernetes.io/projected/b55af1b4-7a9b-4a6f-8801-481e67895f4a-kube-api-access-lqkvk\") pod \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\" (UID: \"b55af1b4-7a9b-4a6f-8801-481e67895f4a\") "
Oct 30 21:37:44.149823 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.149793    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b55af1b4-7a9b-4a6f-8801-481e67895f4a-plugins-config-volume" pod "b55af1b4-7a9b-4a6f-8801-481e67895f4a" (UID: "b55af1b4-7a9b-4a6f-8801-481e67895f4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:44.150192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.150150    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b55af1b4-7a9b-4a6f-8801-481e67895f4a-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:44.152078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.152049    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55af1b4-7a9b-4a6f-8801-481e67895f4a-kube-api-access-lqkvk" pod "b55af1b4-7a9b-4a6f-8801-481e67895f4a" (UID: "b55af1b4-7a9b-4a6f-8801-481e67895f4a"). InnerVolumeSpecName "kube-api-access-lqkvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:44.251434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.251390    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqkvk\" (UniqueName: \"kubernetes.io/projected/b55af1b4-7a9b-4a6f-8801-481e67895f4a-kube-api-access-lqkvk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:44.392187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.392073    5886 generic.go:356] "Generic (PLEG): container finished" podID="b55af1b4-7a9b-4a6f-8801-481e67895f4a" containerID="39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1" exitCode=0
Oct 30 21:37:44.392187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.392121    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"
Oct 30 21:37:44.392187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.392138    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x" event={"ID":"b55af1b4-7a9b-4a6f-8801-481e67895f4a","Type":"ContainerDied","Data":"39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1"}
Oct 30 21:37:44.392470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.392220    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x" event={"ID":"b55af1b4-7a9b-4a6f-8801-481e67895f4a","Type":"ContainerDied","Data":"864816b524186d3731f2d5f7e66a590121efd604ecc4a00a1043b5ef0de4241e"}
Oct 30 21:37:44.392470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.392251    5886 scope.go:122] "RemoveContainer" containerID="39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1"
Oct 30 21:37:44.404903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.404876    5886 scope.go:122] "RemoveContainer" containerID="39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1"
Oct 30 21:37:44.405218 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:44.405198    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1\": container with ID starting with 39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1 not found: ID does not exist" containerID="39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1"
Oct 30 21:37:44.405305 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.405231    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1"} err="failed to get container status \"39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1\": rpc error: code = NotFound desc = could not find container \"39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1\": container with ID starting with 39aa5d7544c268a42546a00c2d7131b158f3351ead954ac05cf62d113b2f29f1 not found: ID does not exist"
Oct 30 21:37:44.441127 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.441091    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"]
Oct 30 21:37:44.455070 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:44.455034    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-26a13f-67b6898c85-cft2x"]
Oct 30 21:37:45.994063 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:45.994022    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b55af1b4-7a9b-4a6f-8801-481e67895f4a" path="/var/lib/kubelet/pods/b55af1b4-7a9b-4a6f-8801-481e67895f4a/volumes"
Oct 30 21:37:47.569590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:47.569550    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"]
Oct 30 21:37:47.570101 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:47.569783    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy" containerID="cri-o://e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872" gracePeriod=30
Oct 30 21:37:48.616822 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:48.616784    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:37:48.617311 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:48.616838    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:37:52.845994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.845920    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:37:52.955709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.955677    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-token\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-token\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.955888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.955751    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-envoy\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.955888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.955797    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-socket\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.955888 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.955854    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-podinfo\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.956057 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.955898    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-data\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.956057 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.955998    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-certs\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.956057 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956037    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-kube-api-access-mv92z\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-kube-api-access-mv92z\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.956261 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956080    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istiod-ca-cert\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.956261 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956112    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-credential-socket\") pod \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\" (UID: \"05864e5f-02e0-4bf2-b5b9-4130a34593dd\") "
Oct 30 21:37:52.956261 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956133    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-socket" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:52.956261 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956148    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-data" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:52.956506 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956473    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-credential-socket" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:52.956598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956494    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istiod-ca-cert" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:52.956657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956596    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-certs" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:52.956714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956694    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:52.956714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956708    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:52.956825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956717    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:52.956825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956726    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:52.956825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.956738    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:52.958129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.958097    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-envoy" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:52.958255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.958138    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-token" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:52.958255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.958145    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-podinfo" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:52.958255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:52.958236    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-kube-api-access-mv92z" pod "05864e5f-02e0-4bf2-b5b9-4130a34593dd" (UID: "05864e5f-02e0-4bf2-b5b9-4130a34593dd"). InnerVolumeSpecName "kube-api-access-mv92z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:53.057434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.057400    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:53.057434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.057429    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv92z\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-kube-api-access-mv92z\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:53.057434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.057440    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:53.057670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.057448    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05864e5f-02e0-4bf2-b5b9-4130a34593dd-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:53.453389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.453354    5886 generic.go:356] "Generic (PLEG): container finished" podID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerID="e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872" exitCode=0
Oct 30 21:37:53.453567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.453403    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"
Oct 30 21:37:53.453567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.453439    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" event={"ID":"05864e5f-02e0-4bf2-b5b9-4130a34593dd","Type":"ContainerDied","Data":"e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872"}
Oct 30 21:37:53.453567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.453499    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls" event={"ID":"05864e5f-02e0-4bf2-b5b9-4130a34593dd","Type":"ContainerDied","Data":"ade17f2c48f7f44e1102c63254710eab080c22132ba91fc25b4e48b008b851d2"}
Oct 30 21:37:53.453567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.453522    5886 scope.go:122] "RemoveContainer" containerID="e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872"
Oct 30 21:37:53.464055 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.464037    5886 scope.go:122] "RemoveContainer" containerID="a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68"
Oct 30 21:37:53.473117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.473099    5886 scope.go:122] "RemoveContainer" containerID="e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872"
Oct 30 21:37:53.473386 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:53.473367    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872\": container with ID starting with e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872 not found: ID does not exist" containerID="e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872"
Oct 30 21:37:53.473467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.473398    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872"} err="failed to get container status \"e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872\": rpc error: code = NotFound desc = could not find container \"e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872\": container with ID starting with e267fce216c52c79218f4083b4cddb89d45fe66088905b4c42719aa11df28872 not found: ID does not exist"
Oct 30 21:37:53.473467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.473422    5886 scope.go:122] "RemoveContainer" containerID="a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68"
Oct 30 21:37:53.473661 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:53.473643    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68\": container with ID starting with a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68 not found: ID does not exist" containerID="a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68"
Oct 30 21:37:53.473704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.473668    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68"} err="failed to get container status \"a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68\": rpc error: code = NotFound desc = could not find container \"a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68\": container with ID starting with a1d264c4bca852e2784afe97cb0694f05b0011f6678741ba675ae84e11754a68 not found: ID does not exist"
Oct 30 21:37:53.514205 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.514179    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"]
Oct 30 21:37:53.525436 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.525413    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-wfnls"]
Oct 30 21:37:53.907277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.907247    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"]
Oct 30 21:37:53.907759 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.907442    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c" podUID="32ed146a-6b47-46f5-8e7d-fafe61761bd1" containerName="maas-api" containerID="cri-o://6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f" gracePeriod=30
Oct 30 21:37:53.989622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:53.989591    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" path="/var/lib/kubelet/pods/05864e5f-02e0-4bf2-b5b9-4130a34593dd/volumes"
Oct 30 21:37:54.188540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.188511    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:37:54.271753 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.271710    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ed146a-6b47-46f5-8e7d-fafe61761bd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ed146a-6b47-46f5-8e7d-fafe61761bd1-openshift-service-ca\") pod \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") "
Oct 30 21:37:54.271946 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.271820    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-maas-api-tls\") pod \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") "
Oct 30 21:37:54.271946 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.271897    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ed146a-6b47-46f5-8e7d-fafe61761bd1-kube-api-access-4qppw\" (UniqueName: \"kubernetes.io/projected/32ed146a-6b47-46f5-8e7d-fafe61761bd1-kube-api-access-4qppw\") pod \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") "
Oct 30 21:37:54.273243 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.273213    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-metrics-certs\") pod \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\" (UID: \"32ed146a-6b47-46f5-8e7d-fafe61761bd1\") "
Oct 30 21:37:54.274825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.274792    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ed146a-6b47-46f5-8e7d-fafe61761bd1-openshift-service-ca" pod "32ed146a-6b47-46f5-8e7d-fafe61761bd1" (UID: "32ed146a-6b47-46f5-8e7d-fafe61761bd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:54.276595 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.276569    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-maas-api-tls" pod "32ed146a-6b47-46f5-8e7d-fafe61761bd1" (UID: "32ed146a-6b47-46f5-8e7d-fafe61761bd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:54.276705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.276680    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ed146a-6b47-46f5-8e7d-fafe61761bd1-kube-api-access-4qppw" pod "32ed146a-6b47-46f5-8e7d-fafe61761bd1" (UID: "32ed146a-6b47-46f5-8e7d-fafe61761bd1"). InnerVolumeSpecName "kube-api-access-4qppw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:54.276891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.276864    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-metrics-certs" pod "32ed146a-6b47-46f5-8e7d-fafe61761bd1" (UID: "32ed146a-6b47-46f5-8e7d-fafe61761bd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:54.374862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.374823    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:54.374862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.374861    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ed146a-6b47-46f5-8e7d-fafe61761bd1-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:54.375149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.374875    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ed146a-6b47-46f5-8e7d-fafe61761bd1-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:54.375149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.374889    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qppw\" (UniqueName: \"kubernetes.io/projected/32ed146a-6b47-46f5-8e7d-fafe61761bd1-kube-api-access-4qppw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:54.438524 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.438435    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"]
Oct 30 21:37:54.439064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439044    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439064    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439086    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b55af1b4-7a9b-4a6f-8801-481e67895f4a" containerName="payload-processing"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439094    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55af1b4-7a9b-4a6f-8801-481e67895f4a" containerName="payload-processing"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439105    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439113    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439146    5886 cpu_manager.go:414] "containerMap: removing container" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439168    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439185    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439192    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439208    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy"
Oct 30 21:37:54.439218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439215    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439227    5886 cpu_manager.go:414] "containerMap: removing container" podUID="e91f8d27-558e-4456-89d5-daf1c1c1ebe6" containerName="maas-api"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439234    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91f8d27-558e-4456-89d5-daf1c1c1ebe6" containerName="maas-api"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439245    5886 cpu_manager.go:414] "containerMap: removing container" podUID="421b5cbd-ab97-449d-8330-85d7b653982e" containerName="payload-processing"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439253    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421b5cbd-ab97-449d-8330-85d7b653982e" containerName="payload-processing"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439264    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b21188ce-77a0-4112-9773-6a1fa0a1f955" containerName="payload-pre-processing"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439271    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21188ce-77a0-4112-9773-6a1fa0a1f955" containerName="payload-pre-processing"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439300    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439306    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439319    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439325    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439336    5886 cpu_manager.go:414] "containerMap: removing container" podUID="32ed146a-6b47-46f5-8e7d-fafe61761bd1" containerName="maas-api"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439344    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ed146a-6b47-46f5-8e7d-fafe61761bd1" containerName="maas-api"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439510    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439524    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439534    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2824625-1259-44b3-beac-44087a9d94ad" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439545    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91f8d27-558e-4456-89d5-daf1c1c1ebe6" containerName="maas-api"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439556    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="421b5cbd-ab97-449d-8330-85d7b653982e" containerName="payload-processing"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439565    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21188ce-77a0-4112-9773-6a1fa0a1f955" containerName="payload-pre-processing"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439576    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439584    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16f82a5-11e5-4c70-996f-a7d5ea36573a" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439594    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ed146a-6b47-46f5-8e7d-fafe61761bd1" containerName="maas-api"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439613    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b55af1b4-7a9b-4a6f-8801-481e67895f4a" containerName="payload-processing"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439623    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="a233a4d2-3e10-4170-ac3f-d11fc6302225" containerName="istio-proxy"
Oct 30 21:37:54.439811 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.439635    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy"
Oct 30 21:37:54.443562 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.443540    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:54.459977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.459947    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4e4693\""
Oct 30 21:37:54.460459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.460436    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"]
Oct 30 21:37:54.462973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.462947    5886 generic.go:356] "Generic (PLEG): container finished" podID="32ed146a-6b47-46f5-8e7d-fafe61761bd1" containerID="6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f" exitCode=0
Oct 30 21:37:54.463285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.463256    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c" event={"ID":"32ed146a-6b47-46f5-8e7d-fafe61761bd1","Type":"ContainerDied","Data":"6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f"}
Oct 30 21:37:54.463397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.463311    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c" event={"ID":"32ed146a-6b47-46f5-8e7d-fafe61761bd1","Type":"ContainerDied","Data":"c1ae3a0bafdb3f84b63b582d50d31da36d184f6ce7669a566804c71ff8336fd9"}
Oct 30 21:37:54.463397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.463343    5886 scope.go:122] "RemoveContainer" containerID="6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f"
Oct 30 21:37:54.463495 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.463457    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"
Oct 30 21:37:54.478262 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.478240    5886 scope.go:122] "RemoveContainer" containerID="6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f"
Oct 30 21:37:54.478589 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:54.478570    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f\": container with ID starting with 6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f not found: ID does not exist" containerID="6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f"
Oct 30 21:37:54.478676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.478595    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f"} err="failed to get container status \"6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f\": rpc error: code = NotFound desc = could not find container \"6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f\": container with ID starting with 6b05f46afe6d0ba19fcefc2932b875161a5c2092d33d5095e31dc6451431fa5f not found: ID does not exist"
Oct 30 21:37:54.520484 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.520454    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"]
Oct 30 21:37:54.531541 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.531508    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f05cda-7d69557bbb-5xs8c"]
Oct 30 21:37:54.577560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.577524    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bbp\" (UniqueName: \"kubernetes.io/projected/5a6c6198-7f2e-4a4f-b180-d44786fa955b-kube-api-access-22bbp\") pod \"payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln\" (UID: \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:54.577812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.577784    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c6198-7f2e-4a4f-b180-d44786fa955b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln\" (UID: \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:54.679108 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.679070    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bbp\" (UniqueName: \"kubernetes.io/projected/5a6c6198-7f2e-4a4f-b180-d44786fa955b-kube-api-access-22bbp\") pod \"payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln\" (UID: \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:54.679340 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.679282    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c6198-7f2e-4a4f-b180-d44786fa955b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln\" (UID: \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:54.680279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.680125    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c6198-7f2e-4a4f-b180-d44786fa955b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln\" (UID: \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:54.698518 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.698453    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bbp\" (UniqueName: \"kubernetes.io/projected/5a6c6198-7f2e-4a4f-b180-d44786fa955b-kube-api-access-22bbp\") pod \"payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln\" (UID: \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:54.758456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:54.758418    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:55.068442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:55.068367    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"]
Oct 30 21:37:55.073010 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:37:55.072976    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6c6198_7f2e_4a4f_b180_d44786fa955b.slice/crio-2c594f5d8ccb5282302b34387e86cb16533e2a5b3b767456dc14a3c3c3beb158 WatchSource:0}: Error finding container 2c594f5d8ccb5282302b34387e86cb16533e2a5b3b767456dc14a3c3c3beb158: Status 404 returned error can't find the container with id 2c594f5d8ccb5282302b34387e86cb16533e2a5b3b767456dc14a3c3c3beb158
Oct 30 21:37:55.474937 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:55.474898    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln" event={"ID":"5a6c6198-7f2e-4a4f-b180-d44786fa955b","Type":"ContainerStarted","Data":"48f1a7ffafb97d3b1fdd64b92ece6b14862c9bdad82412a50f6867ffe6c66d90"}
Oct 30 21:37:55.475150 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:55.474959    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln" event={"ID":"5a6c6198-7f2e-4a4f-b180-d44786fa955b","Type":"ContainerStarted","Data":"2c594f5d8ccb5282302b34387e86cb16533e2a5b3b767456dc14a3c3c3beb158"}
Oct 30 21:37:55.475150 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:55.475041    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:37:55.510967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:55.510915    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln" podStartSLOduration=1.510901249 podStartE2EDuration="1.510901249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:55.510779033 +0000 UTC m=+1675.640030490" watchObservedRunningTime="2026-10-30 21:37:55.510901249 +0000 UTC m=+1675.640152728"
Oct 30 21:37:55.995190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:55.995121    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ed146a-6b47-46f5-8e7d-fafe61761bd1" path="/var/lib/kubelet/pods/32ed146a-6b47-46f5-8e7d-fafe61761bd1/volumes"
Oct 30 21:37:56.365049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.364943    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"]
Oct 30 21:37:56.365602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.365210    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj" podUID="510a0bd4-b176-4497-b22a-0b73cd039fcc" containerName="maas-api" containerID="cri-o://cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a" gracePeriod=30
Oct 30 21:37:56.641064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.640994    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:37:56.803893 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.803699    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-maas-api-tls\") pod \"510a0bd4-b176-4497-b22a-0b73cd039fcc\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") "
Oct 30 21:37:56.803893 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.803868    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-metrics-certs\") pod \"510a0bd4-b176-4497-b22a-0b73cd039fcc\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") "
Oct 30 21:37:56.804118 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.803942    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510a0bd4-b176-4497-b22a-0b73cd039fcc-kube-api-access-94k9h\" (UniqueName: \"kubernetes.io/projected/510a0bd4-b176-4497-b22a-0b73cd039fcc-kube-api-access-94k9h\") pod \"510a0bd4-b176-4497-b22a-0b73cd039fcc\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") "
Oct 30 21:37:56.804118 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.803977    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510a0bd4-b176-4497-b22a-0b73cd039fcc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a0bd4-b176-4497-b22a-0b73cd039fcc-openshift-service-ca\") pod \"510a0bd4-b176-4497-b22a-0b73cd039fcc\" (UID: \"510a0bd4-b176-4497-b22a-0b73cd039fcc\") "
Oct 30 21:37:56.804696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.804667    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510a0bd4-b176-4497-b22a-0b73cd039fcc-openshift-service-ca" pod "510a0bd4-b176-4497-b22a-0b73cd039fcc" (UID: "510a0bd4-b176-4497-b22a-0b73cd039fcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:56.807609 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.807584    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-maas-api-tls" pod "510a0bd4-b176-4497-b22a-0b73cd039fcc" (UID: "510a0bd4-b176-4497-b22a-0b73cd039fcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:56.808370 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.807948    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510a0bd4-b176-4497-b22a-0b73cd039fcc-kube-api-access-94k9h" pod "510a0bd4-b176-4497-b22a-0b73cd039fcc" (UID: "510a0bd4-b176-4497-b22a-0b73cd039fcc"). InnerVolumeSpecName "kube-api-access-94k9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:56.808877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.808837    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-metrics-certs" pod "510a0bd4-b176-4497-b22a-0b73cd039fcc" (UID: "510a0bd4-b176-4497-b22a-0b73cd039fcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:56.824313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.824285    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"]
Oct 30 21:37:56.824735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.824682    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf" podUID="6943443c-50b8-4113-8c1b-88ef3b2f057c" containerName="maas-api" containerID="cri-o://61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c" gracePeriod=30
Oct 30 21:37:56.905422 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.905389    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:56.905422 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.905420    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94k9h\" (UniqueName: \"kubernetes.io/projected/510a0bd4-b176-4497-b22a-0b73cd039fcc-kube-api-access-94k9h\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:56.905422 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.905430    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a0bd4-b176-4497-b22a-0b73cd039fcc-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:56.905745 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:56.905439    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/510a0bd4-b176-4497-b22a-0b73cd039fcc-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:57.076666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.076639    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:37:57.209376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.209288    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6943443c-50b8-4113-8c1b-88ef3b2f057c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6943443c-50b8-4113-8c1b-88ef3b2f057c-openshift-service-ca\") pod \"6943443c-50b8-4113-8c1b-88ef3b2f057c\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") "
Oct 30 21:37:57.209558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.209424    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-metrics-certs\") pod \"6943443c-50b8-4113-8c1b-88ef3b2f057c\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") "
Oct 30 21:37:57.209558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.209499    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-maas-api-tls\") pod \"6943443c-50b8-4113-8c1b-88ef3b2f057c\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") "
Oct 30 21:37:57.209558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.209546    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6943443c-50b8-4113-8c1b-88ef3b2f057c-kube-api-access-j8s8j\" (UniqueName: \"kubernetes.io/projected/6943443c-50b8-4113-8c1b-88ef3b2f057c-kube-api-access-j8s8j\") pod \"6943443c-50b8-4113-8c1b-88ef3b2f057c\" (UID: \"6943443c-50b8-4113-8c1b-88ef3b2f057c\") "
Oct 30 21:37:57.209719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.209689    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6943443c-50b8-4113-8c1b-88ef3b2f057c-openshift-service-ca" pod "6943443c-50b8-4113-8c1b-88ef3b2f057c" (UID: "6943443c-50b8-4113-8c1b-88ef3b2f057c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:57.209982 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.209950    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6943443c-50b8-4113-8c1b-88ef3b2f057c-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:57.211636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.211607    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6943443c-50b8-4113-8c1b-88ef3b2f057c-kube-api-access-j8s8j" pod "6943443c-50b8-4113-8c1b-88ef3b2f057c" (UID: "6943443c-50b8-4113-8c1b-88ef3b2f057c"). InnerVolumeSpecName "kube-api-access-j8s8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:57.211636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.211621    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-metrics-certs" pod "6943443c-50b8-4113-8c1b-88ef3b2f057c" (UID: "6943443c-50b8-4113-8c1b-88ef3b2f057c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:57.211759 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.211638    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-maas-api-tls" pod "6943443c-50b8-4113-8c1b-88ef3b2f057c" (UID: "6943443c-50b8-4113-8c1b-88ef3b2f057c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:57.312699 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.312019    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:57.312699 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.312056    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6943443c-50b8-4113-8c1b-88ef3b2f057c-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:57.312699 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.312073    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8s8j\" (UniqueName: \"kubernetes.io/projected/6943443c-50b8-4113-8c1b-88ef3b2f057c-kube-api-access-j8s8j\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:37:57.495853 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.495764    5886 generic.go:356] "Generic (PLEG): container finished" podID="510a0bd4-b176-4497-b22a-0b73cd039fcc" containerID="cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a" exitCode=0
Oct 30 21:37:57.495853 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.495810    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"
Oct 30 21:37:57.496396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.495850    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj" event={"ID":"510a0bd4-b176-4497-b22a-0b73cd039fcc","Type":"ContainerDied","Data":"cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a"}
Oct 30 21:37:57.496396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.495911    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj" event={"ID":"510a0bd4-b176-4497-b22a-0b73cd039fcc","Type":"ContainerDied","Data":"5bf38a4e02a426fe2ed435da83bada4c543d55dbb9708cf7ba3822786d1b7c6b"}
Oct 30 21:37:57.496396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.495928    5886 scope.go:122] "RemoveContainer" containerID="cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a"
Oct 30 21:37:57.497522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.497501    5886 generic.go:356] "Generic (PLEG): container finished" podID="6943443c-50b8-4113-8c1b-88ef3b2f057c" containerID="61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c" exitCode=0
Oct 30 21:37:57.497638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.497539    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"
Oct 30 21:37:57.497638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.497541    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf" event={"ID":"6943443c-50b8-4113-8c1b-88ef3b2f057c","Type":"ContainerDied","Data":"61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c"}
Oct 30 21:37:57.497714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.497671    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf" event={"ID":"6943443c-50b8-4113-8c1b-88ef3b2f057c","Type":"ContainerDied","Data":"aaf2d26bda706021f821cffed592e37bf63c0d5ac2d0ba2effece59fb9ec7034"}
Oct 30 21:37:57.507135 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.507115    5886 scope.go:122] "RemoveContainer" containerID="cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a"
Oct 30 21:37:57.507455 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:57.507428    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a\": container with ID starting with cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a not found: ID does not exist" containerID="cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a"
Oct 30 21:37:57.507538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.507461    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a"} err="failed to get container status \"cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a\": rpc error: code = NotFound desc = could not find container \"cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a\": container with ID starting with cec14ff766458e6c710dd3d10e1d7120d9d5da91cba3187a62079bacade9d74a not found: ID does not exist"
Oct 30 21:37:57.507538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.507479    5886 scope.go:122] "RemoveContainer" containerID="61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c"
Oct 30 21:37:57.516591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.516576    5886 scope.go:122] "RemoveContainer" containerID="61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c"
Oct 30 21:37:57.516838 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:37:57.516819    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c\": container with ID starting with 61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c not found: ID does not exist" containerID="61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c"
Oct 30 21:37:57.516887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.516844    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c"} err="failed to get container status \"61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c\": rpc error: code = NotFound desc = could not find container \"61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c\": container with ID starting with 61ce88cd6af79e48c70033d7cda4209d36b430c9b6557aaf4b04f88945dd9d1c not found: ID does not exist"
Oct 30 21:37:57.544608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.544585    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"]
Oct 30 21:37:57.555711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.555687    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-13658f-c7b5fdb47-qx4gj"]
Oct 30 21:37:57.587742 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.587713    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"]
Oct 30 21:37:57.591549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.591514    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-26a13f-86d75b8d79-2qcbf"]
Oct 30 21:37:57.987724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.987687    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510a0bd4-b176-4497-b22a-0b73cd039fcc" path="/var/lib/kubelet/pods/510a0bd4-b176-4497-b22a-0b73cd039fcc/volumes"
Oct 30 21:37:57.988229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:37:57.988208    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6943443c-50b8-4113-8c1b-88ef3b2f057c" path="/var/lib/kubelet/pods/6943443c-50b8-4113-8c1b-88ef3b2f057c/volumes"
Oct 30 21:38:06.491501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:06.491459    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:38:11.029514 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:38:11.029477    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:38:11.886987 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.886950    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"]
Oct 30 21:38:11.887631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887604    5886 cpu_manager.go:414] "containerMap: removing container" podUID="510a0bd4-b176-4497-b22a-0b73cd039fcc" containerName="maas-api"
Oct 30 21:38:11.887631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887628    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510a0bd4-b176-4497-b22a-0b73cd039fcc" containerName="maas-api"
Oct 30 21:38:11.887891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887661    5886 cpu_manager.go:414] "containerMap: removing container" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy"
Oct 30 21:38:11.887891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887669    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05864e5f-02e0-4bf2-b5b9-4130a34593dd" containerName="istio-proxy"
Oct 30 21:38:11.887891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887711    5886 cpu_manager.go:414] "containerMap: removing container" podUID="6943443c-50b8-4113-8c1b-88ef3b2f057c" containerName="maas-api"
Oct 30 21:38:11.887891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887717    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6943443c-50b8-4113-8c1b-88ef3b2f057c" containerName="maas-api"
Oct 30 21:38:11.887891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887801    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="510a0bd4-b176-4497-b22a-0b73cd039fcc" containerName="maas-api"
Oct 30 21:38:11.887891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887815    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="2930b56b-c0fe-46a3-a059-04de254fb9f8" containerName="istio-proxy"
Oct 30 21:38:11.887891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.887825    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="6943443c-50b8-4113-8c1b-88ef3b2f057c" containerName="maas-api"
Oct 30 21:38:11.893059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.893039    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:11.913630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.913436    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"]
Oct 30 21:38:11.921145 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.921118    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4e4693-serving-cert\""
Oct 30 21:38:11.986428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.986395    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:11.986630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.986446    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-metrics-certs\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:11.986630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.986491    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4rm\" (UniqueName: \"kubernetes.io/projected/3a782cd5-ce3f-4709-b343-4a15776d3a76-kube-api-access-4g4rm\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:11.986788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:11.986631    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a782cd5-ce3f-4709-b343-4a15776d3a76-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.087914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.087881    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.088435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.087929    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-metrics-certs\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.088435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.087966    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4rm\" (UniqueName: \"kubernetes.io/projected/3a782cd5-ce3f-4709-b343-4a15776d3a76-kube-api-access-4g4rm\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.088435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.088015    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a782cd5-ce3f-4709-b343-4a15776d3a76-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.089360 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.089330    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a782cd5-ce3f-4709-b343-4a15776d3a76-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.091020 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.090998    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.091133 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.091106    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-metrics-certs\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.106251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.106225    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4rm\" (UniqueName: \"kubernetes.io/projected/3a782cd5-ce3f-4709-b343-4a15776d3a76-kube-api-access-4g4rm\") pod \"maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.216693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.216656    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:12.521057 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.520970    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"]
Oct 30 21:38:12.525482 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:38:12.525448    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a782cd5_ce3f_4709_b343_4a15776d3a76.slice/crio-f02e3d6352f9fe27214f0ba911124a96278d45c2a32bc17831293c4518110102 WatchSource:0}: Error finding container f02e3d6352f9fe27214f0ba911124a96278d45c2a32bc17831293c4518110102: Status 404 returned error can't find the container with id f02e3d6352f9fe27214f0ba911124a96278d45c2a32bc17831293c4518110102
Oct 30 21:38:12.526948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.526926    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:38:12.601395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:12.601358    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz" event={"ID":"3a782cd5-ce3f-4709-b343-4a15776d3a76","Type":"ContainerStarted","Data":"f02e3d6352f9fe27214f0ba911124a96278d45c2a32bc17831293c4518110102"}
Oct 30 21:38:13.611900 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:13.611857    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz" event={"ID":"3a782cd5-ce3f-4709-b343-4a15776d3a76","Type":"ContainerStarted","Data":"7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be"}
Oct 30 21:38:13.612323 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:13.611979    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:13.668437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:13.668375    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz" podStartSLOduration=2.25991618 podStartE2EDuration="2.668358376s" totalImagesPullingTime="408.442196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:13.667970812 +0000 UTC m=+1693.797222271" watchObservedRunningTime="2026-10-30 21:38:13.668358376 +0000 UTC m=+1693.797609833"
Oct 30 21:38:19.624051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:19.624023    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:38:20.393241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.393179    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"]
Oct 30 21:38:20.393480 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.393445    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8" podUID="34928eaa-d2c3-4bd4-8f90-2553a7e538aa" containerName="maas-api" containerID="cri-o://39d99e22afb1bea2d08e88051457a96944fc661fdf12562e17b2d2df28c7c4aa" gracePeriod=30
Oct 30 21:38:20.669885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.669461    5886 generic.go:356] "Generic (PLEG): container finished" podID="34928eaa-d2c3-4bd4-8f90-2553a7e538aa" containerID="39d99e22afb1bea2d08e88051457a96944fc661fdf12562e17b2d2df28c7c4aa" exitCode=0
Oct 30 21:38:20.669885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.669695    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8" event={"ID":"34928eaa-d2c3-4bd4-8f90-2553a7e538aa","Type":"ContainerDied","Data":"39d99e22afb1bea2d08e88051457a96944fc661fdf12562e17b2d2df28c7c4aa"}
Oct 30 21:38:20.669885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.669748    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8" event={"ID":"34928eaa-d2c3-4bd4-8f90-2553a7e538aa","Type":"ContainerDied","Data":"5924b8eafd4dc40c60ef3ed87b202dd68c515087cd9bb6af01b07ca475e0e630"}
Oct 30 21:38:20.669885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.669775    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5924b8eafd4dc40c60ef3ed87b202dd68c515087cd9bb6af01b07ca475e0e630"
Oct 30 21:38:20.679031 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.678998    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:38:20.794498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.791939    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-maas-api-tls\") pod \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") "
Oct 30 21:38:20.794498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.792044    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-metrics-certs\") pod \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") "
Oct 30 21:38:20.794498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.792077    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-kube-api-access-q7j9c\" (UniqueName: \"kubernetes.io/projected/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-kube-api-access-q7j9c\") pod \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") "
Oct 30 21:38:20.794498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.792150    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-openshift-service-ca\") pod \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\" (UID: \"34928eaa-d2c3-4bd4-8f90-2553a7e538aa\") "
Oct 30 21:38:20.794498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.792699    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-openshift-service-ca" pod "34928eaa-d2c3-4bd4-8f90-2553a7e538aa" (UID: "34928eaa-d2c3-4bd4-8f90-2553a7e538aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:20.794899 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.794868    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-maas-api-tls" pod "34928eaa-d2c3-4bd4-8f90-2553a7e538aa" (UID: "34928eaa-d2c3-4bd4-8f90-2553a7e538aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:20.795294 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.795260    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-metrics-certs" pod "34928eaa-d2c3-4bd4-8f90-2553a7e538aa" (UID: "34928eaa-d2c3-4bd4-8f90-2553a7e538aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:20.798485 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.798458    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-kube-api-access-q7j9c" pod "34928eaa-d2c3-4bd4-8f90-2553a7e538aa" (UID: "34928eaa-d2c3-4bd4-8f90-2553a7e538aa"). InnerVolumeSpecName "kube-api-access-q7j9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:20.899025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.898987    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:20.899025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.899026    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7j9c\" (UniqueName: \"kubernetes.io/projected/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-kube-api-access-q7j9c\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:20.899335 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.899042    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:20.899335 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:20.899056    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34928eaa-d2c3-4bd4-8f90-2553a7e538aa-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:21.677732 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:21.677697    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"
Oct 30 21:38:21.728817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:21.728775    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"]
Oct 30 21:38:21.739928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:21.739892    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ea45c0-7df6bf445-4lzl8"]
Oct 30 21:38:21.989428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:21.989354    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34928eaa-d2c3-4bd4-8f90-2553a7e538aa" path="/var/lib/kubelet/pods/34928eaa-d2c3-4bd4-8f90-2553a7e538aa/volumes"
Oct 30 21:38:24.118624 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.118583    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"]
Oct 30 21:38:24.119121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.118900    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc" podUID="ec32f4e7-1385-49af-af57-653322f0d878" containerName="maas-api" containerID="cri-o://cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1" gracePeriod=30
Oct 30 21:38:24.402726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.402695    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:38:24.546887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.546856    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-maas-api-tls\") pod \"ec32f4e7-1385-49af-af57-653322f0d878\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") "
Oct 30 21:38:24.547049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.546945    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec32f4e7-1385-49af-af57-653322f0d878-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec32f4e7-1385-49af-af57-653322f0d878-openshift-service-ca\") pod \"ec32f4e7-1385-49af-af57-653322f0d878\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") "
Oct 30 21:38:24.547092 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.547065    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec32f4e7-1385-49af-af57-653322f0d878-kube-api-access-g4fmg\" (UniqueName: \"kubernetes.io/projected/ec32f4e7-1385-49af-af57-653322f0d878-kube-api-access-g4fmg\") pod \"ec32f4e7-1385-49af-af57-653322f0d878\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") "
Oct 30 21:38:24.547137 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.547095    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-metrics-certs\") pod \"ec32f4e7-1385-49af-af57-653322f0d878\" (UID: \"ec32f4e7-1385-49af-af57-653322f0d878\") "
Oct 30 21:38:24.547335 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.547310    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec32f4e7-1385-49af-af57-653322f0d878-openshift-service-ca" pod "ec32f4e7-1385-49af-af57-653322f0d878" (UID: "ec32f4e7-1385-49af-af57-653322f0d878"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:24.547536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.547512    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec32f4e7-1385-49af-af57-653322f0d878-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:24.548946 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.548921    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-maas-api-tls" pod "ec32f4e7-1385-49af-af57-653322f0d878" (UID: "ec32f4e7-1385-49af-af57-653322f0d878"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:24.549041 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.549031    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-metrics-certs" pod "ec32f4e7-1385-49af-af57-653322f0d878" (UID: "ec32f4e7-1385-49af-af57-653322f0d878"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:24.549195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.549134    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec32f4e7-1385-49af-af57-653322f0d878-kube-api-access-g4fmg" pod "ec32f4e7-1385-49af-af57-653322f0d878" (UID: "ec32f4e7-1385-49af-af57-653322f0d878"). InnerVolumeSpecName "kube-api-access-g4fmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:24.648378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.648349    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4fmg\" (UniqueName: \"kubernetes.io/projected/ec32f4e7-1385-49af-af57-653322f0d878-kube-api-access-g4fmg\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:24.648378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.648378    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:24.648590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.648388    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec32f4e7-1385-49af-af57-653322f0d878-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:24.708263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.708225    5886 generic.go:356] "Generic (PLEG): container finished" podID="ec32f4e7-1385-49af-af57-653322f0d878" containerID="cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1" exitCode=0
Oct 30 21:38:24.708459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.708277    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"
Oct 30 21:38:24.708459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.708314    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc" event={"ID":"ec32f4e7-1385-49af-af57-653322f0d878","Type":"ContainerDied","Data":"cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1"}
Oct 30 21:38:24.708459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.708374    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc" event={"ID":"ec32f4e7-1385-49af-af57-653322f0d878","Type":"ContainerDied","Data":"4944b588fe5fa8f2aceef9a4220b935e2e929cf03c88c5b7cdfa22ef54397cd5"}
Oct 30 21:38:24.708459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.708399    5886 scope.go:122] "RemoveContainer" containerID="cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1"
Oct 30 21:38:24.719261 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.719127    5886 scope.go:122] "RemoveContainer" containerID="cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1"
Oct 30 21:38:24.719440 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:38:24.719420    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1\": container with ID starting with cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1 not found: ID does not exist" containerID="cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1"
Oct 30 21:38:24.719499 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.719448    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1"} err="failed to get container status \"cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1\": rpc error: code = NotFound desc = could not find container \"cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1\": container with ID starting with cee0e50b335fb682c7846e38c35e152cac45765cb99c58850071f79fdc6c73e1 not found: ID does not exist"
Oct 30 21:38:24.760228 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.760194    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"]
Oct 30 21:38:24.771306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:24.771275    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8a20c6-556b98d759-9spbc"]
Oct 30 21:38:25.988907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:25.988876    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec32f4e7-1385-49af-af57-653322f0d878" path="/var/lib/kubelet/pods/ec32f4e7-1385-49af-af57-653322f0d878/volumes"
Oct 30 21:38:41.029499 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:38:41.029411    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:38:50.621826 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.621784    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"]
Oct 30 21:38:50.622331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.622007    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv" podUID="2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" containerName="maas-api" containerID="cri-o://020729da56e5c430b0e17b4d8a4a90910e20eca3dbef24d87725211437cf1304" gracePeriod=30
Oct 30 21:38:50.912169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.912097    5886 generic.go:356] "Generic (PLEG): container finished" podID="2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" containerID="020729da56e5c430b0e17b4d8a4a90910e20eca3dbef24d87725211437cf1304" exitCode=0
Oct 30 21:38:50.912522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.912267    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv" event={"ID":"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8","Type":"ContainerDied","Data":"020729da56e5c430b0e17b4d8a4a90910e20eca3dbef24d87725211437cf1304"}
Oct 30 21:38:50.949934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.949908    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:38:50.962842 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.962802    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-kube-api-access-rxx77\" (UniqueName: \"kubernetes.io/projected/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-kube-api-access-rxx77\") pod \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") "
Oct 30 21:38:50.962977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.962875    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-maas-api-tls\") pod \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") "
Oct 30 21:38:50.962977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.962931    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-openshift-service-ca\") pod \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") "
Oct 30 21:38:50.963482 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.963450    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-openshift-service-ca" pod "2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" (UID: "2757d92d-7ca1-4159-91a5-9f98b9bcbdf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:50.965598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.965565    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-maas-api-tls" pod "2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" (UID: "2757d92d-7ca1-4159-91a5-9f98b9bcbdf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:50.965598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:50.965578    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-kube-api-access-rxx77" pod "2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" (UID: "2757d92d-7ca1-4159-91a5-9f98b9bcbdf8"). InnerVolumeSpecName "kube-api-access-rxx77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:51.017407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.017371    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"]
Oct 30 21:38:51.017603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.017579    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2" podUID="6e514d66-470e-4220-98fb-71057b79cf0a" containerName="maas-api" containerID="cri-o://07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0" gracePeriod=30
Oct 30 21:38:51.064612 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.064575    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-metrics-certs\") pod \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\" (UID: \"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8\") "
Oct 30 21:38:51.065001 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.064980    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxx77\" (UniqueName: \"kubernetes.io/projected/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-kube-api-access-rxx77\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.065096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.065007    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.065096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.065025    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.067438 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.067406    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-metrics-certs" pod "2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" (UID: "2757d92d-7ca1-4159-91a5-9f98b9bcbdf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:51.168378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.166469    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.280121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.280092    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:38:51.368733 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.368699    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e514d66-470e-4220-98fb-71057b79cf0a-kube-api-access-6t4lx\" (UniqueName: \"kubernetes.io/projected/6e514d66-470e-4220-98fb-71057b79cf0a-kube-api-access-6t4lx\") pod \"6e514d66-470e-4220-98fb-71057b79cf0a\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") "
Oct 30 21:38:51.368948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.368825    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-maas-api-tls\") pod \"6e514d66-470e-4220-98fb-71057b79cf0a\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") "
Oct 30 21:38:51.368948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.368884    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-metrics-certs\") pod \"6e514d66-470e-4220-98fb-71057b79cf0a\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") "
Oct 30 21:38:51.368948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.368905    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e514d66-470e-4220-98fb-71057b79cf0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e514d66-470e-4220-98fb-71057b79cf0a-openshift-service-ca\") pod \"6e514d66-470e-4220-98fb-71057b79cf0a\" (UID: \"6e514d66-470e-4220-98fb-71057b79cf0a\") "
Oct 30 21:38:51.369600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.369558    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e514d66-470e-4220-98fb-71057b79cf0a-openshift-service-ca" pod "6e514d66-470e-4220-98fb-71057b79cf0a" (UID: "6e514d66-470e-4220-98fb-71057b79cf0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:51.371570 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.371542    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-metrics-certs" pod "6e514d66-470e-4220-98fb-71057b79cf0a" (UID: "6e514d66-470e-4220-98fb-71057b79cf0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:51.371828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.371799    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-maas-api-tls" pod "6e514d66-470e-4220-98fb-71057b79cf0a" (UID: "6e514d66-470e-4220-98fb-71057b79cf0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:51.372215 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.372196    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e514d66-470e-4220-98fb-71057b79cf0a-kube-api-access-6t4lx" pod "6e514d66-470e-4220-98fb-71057b79cf0a" (UID: "6e514d66-470e-4220-98fb-71057b79cf0a"). InnerVolumeSpecName "kube-api-access-6t4lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:51.472183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.472022    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.472183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.472059    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e514d66-470e-4220-98fb-71057b79cf0a-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.472183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.472078    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t4lx\" (UniqueName: \"kubernetes.io/projected/6e514d66-470e-4220-98fb-71057b79cf0a-kube-api-access-6t4lx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.472183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.472094    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e514d66-470e-4220-98fb-71057b79cf0a-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:38:51.922798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.922769    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"
Oct 30 21:38:51.923315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.922765    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv" event={"ID":"2757d92d-7ca1-4159-91a5-9f98b9bcbdf8","Type":"ContainerDied","Data":"62ae3feca7e71c16ef07d8bd3534ac40716b94a7febc577ad3835d39db96950a"}
Oct 30 21:38:51.923315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.922977    5886 scope.go:122] "RemoveContainer" containerID="020729da56e5c430b0e17b4d8a4a90910e20eca3dbef24d87725211437cf1304"
Oct 30 21:38:51.924858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.924693    5886 generic.go:356] "Generic (PLEG): container finished" podID="6e514d66-470e-4220-98fb-71057b79cf0a" containerID="07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0" exitCode=0
Oct 30 21:38:51.924929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.924900    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2" event={"ID":"6e514d66-470e-4220-98fb-71057b79cf0a","Type":"ContainerDied","Data":"07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0"}
Oct 30 21:38:51.924990 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.924947    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2" event={"ID":"6e514d66-470e-4220-98fb-71057b79cf0a","Type":"ContainerDied","Data":"31aafa65fe54c1fdf43f6f41cd1dee3e5ed1ad50f96735de71a10c8137be141e"}
Oct 30 21:38:51.925072 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.925054    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"
Oct 30 21:38:51.936647 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.936603    5886 scope.go:122] "RemoveContainer" containerID="07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0"
Oct 30 21:38:51.946645 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.946627    5886 scope.go:122] "RemoveContainer" containerID="07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0"
Oct 30 21:38:51.946912 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:38:51.946886    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0\": container with ID starting with 07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0 not found: ID does not exist" containerID="07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0"
Oct 30 21:38:51.946996 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.946916    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0"} err="failed to get container status \"07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0\": rpc error: code = NotFound desc = could not find container \"07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0\": container with ID starting with 07a0a863e8886a7f5a4ff79a9459d79fada669a619eb17ad34aa9e2f7b39c1c0 not found: ID does not exist"
Oct 30 21:38:51.971861 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.971840    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"]
Oct 30 21:38:51.987887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:51.987866    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0e991-666958c99-l98rv"]
Oct 30 21:38:52.021294 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:52.021263    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"]
Oct 30 21:38:52.023968 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:52.023948    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-912e61-6b9598d4b6-2s4k2"]
Oct 30 21:38:53.989259 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:53.989223    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" path="/var/lib/kubelet/pods/2757d92d-7ca1-4159-91a5-9f98b9bcbdf8/volumes"
Oct 30 21:38:53.989757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:53.989708    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e514d66-470e-4220-98fb-71057b79cf0a" path="/var/lib/kubelet/pods/6e514d66-470e-4220-98fb-71057b79cf0a/volumes"
Oct 30 21:38:57.645349 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645316    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"]
Oct 30 21:38:57.645862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645846    5886 cpu_manager.go:414] "containerMap: removing container" podUID="6e514d66-470e-4220-98fb-71057b79cf0a" containerName="maas-api"
Oct 30 21:38:57.645862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645862    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e514d66-470e-4220-98fb-71057b79cf0a" containerName="maas-api"
Oct 30 21:38:57.645960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645881    5886 cpu_manager.go:414] "containerMap: removing container" podUID="34928eaa-d2c3-4bd4-8f90-2553a7e538aa" containerName="maas-api"
Oct 30 21:38:57.645960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645886    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34928eaa-d2c3-4bd4-8f90-2553a7e538aa" containerName="maas-api"
Oct 30 21:38:57.645960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645905    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" containerName="maas-api"
Oct 30 21:38:57.645960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645910    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" containerName="maas-api"
Oct 30 21:38:57.645960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645916    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ec32f4e7-1385-49af-af57-653322f0d878" containerName="maas-api"
Oct 30 21:38:57.645960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645921    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec32f4e7-1385-49af-af57-653322f0d878" containerName="maas-api"
Oct 30 21:38:57.646139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.645992    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e514d66-470e-4220-98fb-71057b79cf0a" containerName="maas-api"
Oct 30 21:38:57.646139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.646003    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="34928eaa-d2c3-4bd4-8f90-2553a7e538aa" containerName="maas-api"
Oct 30 21:38:57.646139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.646012    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="2757d92d-7ca1-4159-91a5-9f98b9bcbdf8" containerName="maas-api"
Oct 30 21:38:57.646139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.646018    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec32f4e7-1385-49af-af57-653322f0d878" containerName="maas-api"
Oct 30 21:38:57.650963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.650942    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.665760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.665738    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"]
Oct 30 21:38:57.669832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.669811    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.674278 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.674254    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b31518-openshift-default-dockercfg-xdgb8\""
Oct 30 21:38:57.679929 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.679507    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"]
Oct 30 21:38:57.709252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.709221    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"]
Oct 30 21:38:57.714252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.714222    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"]
Oct 30 21:38:57.714391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.714339    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.730270 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.730243    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"]
Oct 30 21:38:57.765469 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765429    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.765637 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765486    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.765637 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765542    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15707226-8100-4108-98f0-8697986f92af-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.765770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765736    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.765830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765801    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.765889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765839    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f00c30-1797-45cb-a70e-89188f259de4-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.765889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765870    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcdsb\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-kube-api-access-pcdsb\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.765963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765935    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.766007 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.765988    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.766091 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766076    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.766169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766143    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.766231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766190    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f00c30-1797-45cb-a70e-89188f259de4-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.766274 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766242    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15707226-8100-4108-98f0-8697986f92af-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.766308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766274    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.766342 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766316    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rft9l\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-kube-api-access-rft9l\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.766407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766393    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.766446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766427    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.766490 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.766443    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.867994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.867953    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.868188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868056    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.868188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868101    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15707226-8100-4108-98f0-8697986f92af-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.868188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868134    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.868188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868185    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.868426 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868225    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.868426 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868302    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f00c30-1797-45cb-a70e-89188f259de4-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.868426 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868362    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdsb\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-kube-api-access-pcdsb\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.868426 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868390    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd02068-a0b4-44b0-905c-bd4e0e650977-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.868426 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868421    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.868677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868457    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.868677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868460    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.868677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868495    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.868677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868558    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-kube-api-access-vk8tq\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.868677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868596    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.868677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868623    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.868677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868641    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868682    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868744    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868771    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868800    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868874    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868918    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868951    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f00c30-1797-45cb-a70e-89188f259de4-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868961    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.868973    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15707226-8100-4108-98f0-8697986f92af-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869017    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869046    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rft9l\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-kube-api-access-rft9l\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869080    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869127    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869194    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869261    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869594    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869658    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15707226-8100-4108-98f0-8697986f92af-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.869796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869760    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.870945 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.869930    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.870945 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.870587    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f00c30-1797-45cb-a70e-89188f259de4-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.871474 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.871450    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15707226-8100-4108-98f0-8697986f92af-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.872477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.872452    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f00c30-1797-45cb-a70e-89188f259de4-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.872745 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.872724    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.872871 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.872849    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.887702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.887671    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.887864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.887779    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdsb\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-kube-api-access-pcdsb\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.888026 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.888004    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.888146 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.888124    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rft9l\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-kube-api-access-rft9l\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.967775 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.967736    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:57.970579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970549    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.970734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970621    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.970734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970672    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd02068-a0b4-44b0-905c-bd4e0e650977-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.970734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970703    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.970909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970774    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-kube-api-access-vk8tq\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.970909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970806    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.970909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970856    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.970909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970881    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.970909 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970904    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.971187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.970991    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-data\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.971515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.971302    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-credential-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.971515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.971339    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-certs\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.971515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.971395    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd02068-a0b4-44b0-905c-bd4e0e650977-istiod-ca-cert\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.971757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.971631    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-socket\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.973782 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.973752    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-envoy\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.974276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.974255    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-podinfo\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.986664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.986633    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:38:57.993839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.993815    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-kube-api-access-vk8tq\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:57.994655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:57.994602    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-token\") pod \"e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") " pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:58.030400 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.029925    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:38:58.295973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.295898    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"]
Oct 30 21:38:58.300093 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:38:58.300066    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15707226_8100_4108_98f0_8697986f92af.slice/crio-10f8cb5b21caf1dae84658c1b5085253264e7b3833687f05e2ca1186dff24344 WatchSource:0}: Error finding container 10f8cb5b21caf1dae84658c1b5085253264e7b3833687f05e2ca1186dff24344: Status 404 returned error can't find the container with id 10f8cb5b21caf1dae84658c1b5085253264e7b3833687f05e2ca1186dff24344
Oct 30 21:38:58.303526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.303067    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.303526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.303164    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.303526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.303200    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.316631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.316599    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"]
Oct 30 21:38:58.320931 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:38:58.320907    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f00c30_1797_45cb_a70e_89188f259de4.slice/crio-ed655e9b5837f3c762dcc0b2a361b6e59dc8331af803c5cb7556e289afcd92d3 WatchSource:0}: Error finding container ed655e9b5837f3c762dcc0b2a361b6e59dc8331af803c5cb7556e289afcd92d3: Status 404 returned error can't find the container with id ed655e9b5837f3c762dcc0b2a361b6e59dc8331af803c5cb7556e289afcd92d3
Oct 30 21:38:58.324288 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.324249    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.324397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.324334    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.324397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.324379    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.352931 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.352909    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"]
Oct 30 21:38:58.356434 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:38:58.356403    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd02068_a0b4_44b0_905c_bd4e0e650977.slice/crio-625fcb64f2470387d8e67b77ab5da8ae0f01f2631cd674cd1eb33af6174210ab WatchSource:0}: Error finding container 625fcb64f2470387d8e67b77ab5da8ae0f01f2631cd674cd1eb33af6174210ab: Status 404 returned error can't find the container with id 625fcb64f2470387d8e67b77ab5da8ae0f01f2631cd674cd1eb33af6174210ab
Oct 30 21:38:58.360097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.359721    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.360097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.359799    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:58.360097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:58.359835    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:38:59.055365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.054399    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" event={"ID":"ccd02068-a0b4-44b0-905c-bd4e0e650977","Type":"ContainerStarted","Data":"7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931"}
Oct 30 21:38:59.056121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.056095    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" event={"ID":"ccd02068-a0b4-44b0-905c-bd4e0e650977","Type":"ContainerStarted","Data":"625fcb64f2470387d8e67b77ab5da8ae0f01f2631cd674cd1eb33af6174210ab"}
Oct 30 21:38:59.057734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.057710    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" event={"ID":"a8f00c30-1797-45cb-a70e-89188f259de4","Type":"ContainerStarted","Data":"df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e"}
Oct 30 21:38:59.057866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.057752    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" event={"ID":"a8f00c30-1797-45cb-a70e-89188f259de4","Type":"ContainerStarted","Data":"ed655e9b5837f3c762dcc0b2a361b6e59dc8331af803c5cb7556e289afcd92d3"}
Oct 30 21:38:59.060652 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.060627    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" event={"ID":"15707226-8100-4108-98f0-8697986f92af","Type":"ContainerStarted","Data":"693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530"}
Oct 30 21:38:59.060765 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.060671    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" event={"ID":"15707226-8100-4108-98f0-8697986f92af","Type":"ContainerStarted","Data":"10f8cb5b21caf1dae84658c1b5085253264e7b3833687f05e2ca1186dff24344"}
Oct 30 21:38:59.121476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.121402    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" podStartSLOduration=2.121378105 podStartE2EDuration="2.121378105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:59.102638653 +0000 UTC m=+1739.231890109" watchObservedRunningTime="2026-10-30 21:38:59.121378105 +0000 UTC m=+1739.250629572"
Oct 30 21:38:59.261089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.261018    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" podStartSLOduration=2.261002353 podStartE2EDuration="2.261002353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:59.157508773 +0000 UTC m=+1739.286760230" watchObservedRunningTime="2026-10-30 21:38:59.261002353 +0000 UTC m=+1739.390253810"
Oct 30 21:38:59.266194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.261382    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" podStartSLOduration=2.2613709269999998 podStartE2EDuration="2.261370927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:59.206357145 +0000 UTC m=+1739.335608604" watchObservedRunningTime="2026-10-30 21:38:59.261370927 +0000 UTC m=+1739.390622386"
Oct 30 21:38:59.971494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.971446    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:38:59.974829 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.974792    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:38:59.975108 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:38:59.975084    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:39:00.014898 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.014845    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:39:00.015234 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.015206    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:39:00.047615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.047579    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 21:39:00.047887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.047856    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 21:39:00.050723 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.050696    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:39:00.050958 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.050943    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:39:01.022180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.971417    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:39:01.022180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.971488    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:39:01.022180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.990389    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:39:01.022180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:00.990440    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:39:01.035669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:01.035534    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 21:39:01.035669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:01.035608    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 21:39:01.996177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:01.993119    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:39:01.996177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:01.993199    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:39:01.996177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:01.993857    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:39:01.996177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:01.993895    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:39:02.049786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:02.047466    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 21:39:02.049786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:02.047522    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 21:39:02.970791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:02.970216    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:39:02.970791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:02.970279    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:39:03.001892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:03.001301    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:39:03.001892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:03.001359    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:39:03.051205 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:03.043799    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 21:39:03.051205 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:03.043855    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 21:39:03.970590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:03.970550    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:39:03.970880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:03.970848    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:39:04.002650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:04.002116    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:39:04.002926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:04.002896    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:39:04.032613 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:04.032574    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 21:39:04.032887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:04.032857    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 21:39:04.968732 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:04.968678    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:39:04.969411 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:04.969376    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:39:04.994649 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:04.994615    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:39:04.994907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:04.994875    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:39:05.031670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:05.031622    5886 patch_prober.go:29] interesting pod/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 21:39:05.031861 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:05.031683    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 21:39:06.534148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.534116    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:39:06.535463 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.535410    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:39:06.536665 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.536633    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:39:06.623436 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.623404    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:39:06.624584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.624563    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:39:06.626452 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.626423    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:39:06.751920 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.751890    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:39:06.752478 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.752403    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:39:06.753427 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:06.753403    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:39:11.029529 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:39:11.029491    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:39:15.480921 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.479260    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"]
Oct 30 21:39:15.622178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.621745    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.633666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.633627    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"]
Oct 30 21:39:15.777322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.763637    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a4be78-7304-47b5-a11e-11954990b452-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.777322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.767261    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.777322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.767362    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.777322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.767419    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.777322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.767576    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.777322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.767607    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a4be78-7304-47b5-a11e-11954990b452-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.777322 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.767691    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.803178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.792405    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdz6x\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-kube-api-access-jdz6x\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.803178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.792523    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897614    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897662    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a4be78-7304-47b5-a11e-11954990b452-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897717    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897740    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdz6x\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-kube-api-access-jdz6x\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897780    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897840    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a4be78-7304-47b5-a11e-11954990b452-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897873    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897926    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.900177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.897958    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.919180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.916743    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.919180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.917056    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.919180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.918149    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a4be78-7304-47b5-a11e-11954990b452-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.919180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.918446    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.948181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.942495    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.967192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.955674    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.989241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.978722    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a4be78-7304-47b5-a11e-11954990b452-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:15.994138 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:15.994079    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:16.006848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.006813    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdz6x\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-kube-api-access-jdz6x\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:16.120600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.120571    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:16.152198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.151522    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"]
Oct 30 21:39:16.309942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.304682    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:16.689578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.685653    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"]
Oct 30 21:39:16.689578 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:39:16.688674    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a4be78_7304_47b5_a11e_11954990b452.slice/crio-a2f117307e025e72adf07a01dbb7836a8a77ea2e794fefd4676f6df45cb8ef8c WatchSource:0}: Error finding container a2f117307e025e72adf07a01dbb7836a8a77ea2e794fefd4676f6df45cb8ef8c: Status 404 returned error can't find the container with id a2f117307e025e72adf07a01dbb7836a8a77ea2e794fefd4676f6df45cb8ef8c
Oct 30 21:39:16.701375 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.693177    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.701375 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.693271    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.701375 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.693303    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:16.859357 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.859306    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"]
Oct 30 21:39:16.870073 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.870042    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:16.889412 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.885174    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c19b75-serving-cert\""
Oct 30 21:39:16.892864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:16.892095    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"]
Oct 30 21:39:17.004047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.003933    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.004047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.003976    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhlqg\" (UniqueName: \"kubernetes.io/projected/cb3d7945-8373-4c0c-992f-8df400d9c6b0-kube-api-access-bhlqg\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.004047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.004022    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3d7945-8373-4c0c-992f-8df400d9c6b0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.004414 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.004057    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-metrics-certs\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.105807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.105765    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3d7945-8373-4c0c-992f-8df400d9c6b0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.105989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.105849    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-metrics-certs\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.106093 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.106077    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.106136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.106108    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlqg\" (UniqueName: \"kubernetes.io/projected/cb3d7945-8373-4c0c-992f-8df400d9c6b0-kube-api-access-bhlqg\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.106559 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:39:17.106532    5886 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-serving-cert: secret "maas-api-e2e-praxis-a-c19b75-serving-cert" not found
Oct 30 21:39:17.106676 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:39:17.106602    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls podName:cb3d7945-8373-4c0c-992f-8df400d9c6b0 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:17.606582807 +0000 UTC m=+1757.735834256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls") pod "maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct" (UID: "cb3d7945-8373-4c0c-992f-8df400d9c6b0") : secret "maas-api-e2e-praxis-a-c19b75-serving-cert" not found
Oct 30 21:39:17.107051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.107023    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3d7945-8373-4c0c-992f-8df400d9c6b0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.109107 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.109082    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-metrics-certs\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.127103 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.127073    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhlqg\" (UniqueName: \"kubernetes.io/projected/cb3d7945-8373-4c0c-992f-8df400d9c6b0-kube-api-access-bhlqg\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.503206 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.503144    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" event={"ID":"19a4be78-7304-47b5-a11e-11954990b452","Type":"ContainerStarted","Data":"98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13"}
Oct 30 21:39:17.503407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.503232    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" event={"ID":"19a4be78-7304-47b5-a11e-11954990b452","Type":"ContainerStarted","Data":"a2f117307e025e72adf07a01dbb7836a8a77ea2e794fefd4676f6df45cb8ef8c"}
Oct 30 21:39:17.612236 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.612193    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.614595 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.614565    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.797481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.797386    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:17.813701 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.813628    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podStartSLOduration=2.813608195 podStartE2EDuration="2.813608195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:17.551000746 +0000 UTC m=+1757.680252228" watchObservedRunningTime="2026-10-30 21:39:17.813608195 +0000 UTC m=+1757.942859654"
Oct 30 21:39:17.817169 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.817131    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"]
Oct 30 21:39:17.828182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.828129    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.842961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.842937    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9d677d-openshift-default-dockercfg-l57n8\""
Oct 30 21:39:17.846199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.846150    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"]
Oct 30 21:39:17.921896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.921856    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.922070 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.921971    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxt4\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-kube-api-access-cpxt4\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.922070 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.922002    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.922266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.922105    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.922266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.922137    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.922266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.922231    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.922441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.922380    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.922496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.922468    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c298e40-0c53-4e35-86ab-7d5801546c66-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:17.922578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:17.922563    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.023659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.023609    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.023832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.023668    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.023832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.023694    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.023832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.023729    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.023832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.023813    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c298e40-0c53-4e35-86ab-7d5801546c66-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.023914    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.023957    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.023991    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxt4\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-kube-api-access-cpxt4\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.024021    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.024051    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.024063    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.024223    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024557 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.024510    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.024699 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.024678    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c298e40-0c53-4e35-86ab-7d5801546c66-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.026471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.026449    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.027186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.027144    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.043975 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.043950    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxt4\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-kube-api-access-cpxt4\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.044799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.044138    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.121129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.121047    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:18.122678 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.122652    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 21:39:18.122807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.122701    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 21:39:18.149507 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.149478    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:18.214223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.214041    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"]
Oct 30 21:39:18.224128 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:39:18.224085    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3d7945_8373_4c0c_992f_8df400d9c6b0.slice/crio-c6d0ccbba6c4a68223f92bc75b0c53187db01682c67351a6a7021c738e4190e9 WatchSource:0}: Error finding container c6d0ccbba6c4a68223f92bc75b0c53187db01682c67351a6a7021c738e4190e9: Status 404 returned error can't find the container with id c6d0ccbba6c4a68223f92bc75b0c53187db01682c67351a6a7021c738e4190e9
Oct 30 21:39:18.512037 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.511986    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct" event={"ID":"cb3d7945-8373-4c0c-992f-8df400d9c6b0","Type":"ContainerStarted","Data":"c6d0ccbba6c4a68223f92bc75b0c53187db01682c67351a6a7021c738e4190e9"}
Oct 30 21:39:18.584785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.584748    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"]
Oct 30 21:39:18.598409 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:39:18.598358    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c298e40_0c53_4e35_86ab_7d5801546c66.slice/crio-8172a6c0e3c02d1df5c24831299a8681f818dad373b063ed52fddc9d37296a55 WatchSource:0}: Error finding container 8172a6c0e3c02d1df5c24831299a8681f818dad373b063ed52fddc9d37296a55: Status 404 returned error can't find the container with id 8172a6c0e3c02d1df5c24831299a8681f818dad373b063ed52fddc9d37296a55
Oct 30 21:39:18.600901 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.600865    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:18.601006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.600944    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:18.601006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:18.600986    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:19.122099 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:19.122061    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 21:39:19.122520 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:19.122115    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 21:39:19.541388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:19.541356    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct" event={"ID":"cb3d7945-8373-4c0c-992f-8df400d9c6b0","Type":"ContainerStarted","Data":"b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d"}
Oct 30 21:39:19.541685 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:19.541674    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:19.543768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:19.543743    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" event={"ID":"7c298e40-0c53-4e35-86ab-7d5801546c66","Type":"ContainerStarted","Data":"e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb"}
Oct 30 21:39:19.543884 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:19.543789    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" event={"ID":"7c298e40-0c53-4e35-86ab-7d5801546c66","Type":"ContainerStarted","Data":"8172a6c0e3c02d1df5c24831299a8681f818dad373b063ed52fddc9d37296a55"}
Oct 30 21:39:19.615180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:19.614703    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct" podStartSLOduration=3.226504689 podStartE2EDuration="3.61468142s" totalImagesPullingTime="388.176731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:19.588327633 +0000 UTC m=+1759.717579091" watchObservedRunningTime="2026-10-30 21:39:19.61468142 +0000 UTC m=+1759.743932879"
Oct 30 21:39:19.643710 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:19.643655    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" podStartSLOduration=2.643639076 podStartE2EDuration="2.643639076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:19.63981239 +0000 UTC m=+1759.769063846" watchObservedRunningTime="2026-10-30 21:39:19.643639076 +0000 UTC m=+1759.772890532"
Oct 30 21:39:20.122818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:20.122783    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 21:39:20.123849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:20.123447    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 21:39:20.150749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:20.150724    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:20.155172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:20.152227    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:39:20.155172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:20.152294    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 21:39:21.122180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.121675    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 21:39:21.122180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.121757    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 21:39:21.150149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.150115    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:39:21.150560 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.150186    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 21:39:21.152678 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.152651    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln" podUID="5a6c6198-7f2e-4a4f-b180-d44786fa955b" containerName="payload-processing" containerID="cri-o://48f1a7ffafb97d3b1fdd64b92ece6b14862c9bdad82412a50f6867ffe6c66d90" gracePeriod=25
Oct 30 21:39:21.638151 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.638103    5886 generic.go:356] "Generic (PLEG): container finished" podID="5a6c6198-7f2e-4a4f-b180-d44786fa955b" containerID="48f1a7ffafb97d3b1fdd64b92ece6b14862c9bdad82412a50f6867ffe6c66d90" exitCode=0
Oct 30 21:39:21.638434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.638209    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln" event={"ID":"5a6c6198-7f2e-4a4f-b180-d44786fa955b","Type":"ContainerDied","Data":"48f1a7ffafb97d3b1fdd64b92ece6b14862c9bdad82412a50f6867ffe6c66d90"}
Oct 30 21:39:21.740716 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.740622    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:39:21.847312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.847236    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6c6198-7f2e-4a4f-b180-d44786fa955b-kube-api-access-22bbp\" (UniqueName: \"kubernetes.io/projected/5a6c6198-7f2e-4a4f-b180-d44786fa955b-kube-api-access-22bbp\") pod \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\" (UID: \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\") "
Oct 30 21:39:21.847773 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.847732    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6c6198-7f2e-4a4f-b180-d44786fa955b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c6198-7f2e-4a4f-b180-d44786fa955b-plugins-config-volume\") pod \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\" (UID: \"5a6c6198-7f2e-4a4f-b180-d44786fa955b\") "
Oct 30 21:39:21.849935 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.849901    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6c6198-7f2e-4a4f-b180-d44786fa955b-plugins-config-volume" pod "5a6c6198-7f2e-4a4f-b180-d44786fa955b" (UID: "5a6c6198-7f2e-4a4f-b180-d44786fa955b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:21.854663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.854479    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6c6198-7f2e-4a4f-b180-d44786fa955b-kube-api-access-22bbp" pod "5a6c6198-7f2e-4a4f-b180-d44786fa955b" (UID: "5a6c6198-7f2e-4a4f-b180-d44786fa955b"). InnerVolumeSpecName "kube-api-access-22bbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:21.951441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.951404    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6c6198-7f2e-4a4f-b180-d44786fa955b-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:39:21.951441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:21.951433    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22bbp\" (UniqueName: \"kubernetes.io/projected/5a6c6198-7f2e-4a4f-b180-d44786fa955b-kube-api-access-22bbp\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:39:22.122214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.121993    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 21:39:22.122214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.122064    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 21:39:22.150421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.150301    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:39:22.150421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.150361    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 21:39:22.647740 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.647706    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"
Oct 30 21:39:22.647740 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.647733    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln" event={"ID":"5a6c6198-7f2e-4a4f-b180-d44786fa955b","Type":"ContainerDied","Data":"2c594f5d8ccb5282302b34387e86cb16533e2a5b3b767456dc14a3c3c3beb158"}
Oct 30 21:39:22.648100 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.647798    5886 scope.go:122] "RemoveContainer" containerID="48f1a7ffafb97d3b1fdd64b92ece6b14862c9bdad82412a50f6867ffe6c66d90"
Oct 30 21:39:22.723240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.723193    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"]
Oct 30 21:39:22.727423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:22.727393    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e4693-8ccb95447-4vjln"]
Oct 30 21:39:23.122788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:23.122685    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Oct 30 21:39:23.122788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:23.122760    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Oct 30 21:39:23.150032 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:23.149998    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:39:23.150251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:23.150055    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 21:39:24.007938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:24.007904    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6c6198-7f2e-4a4f-b180-d44786fa955b" path="/var/lib/kubelet/pods/5a6c6198-7f2e-4a4f-b180-d44786fa955b/volumes"
Oct 30 21:39:24.150048 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:24.150009    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:39:24.150285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:24.150067    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 21:39:24.708672 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:24.708623    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:24.710204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:24.710142    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:24.711198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:24.711148    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:39:25.150430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:25.150387    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:39:25.150859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:25.150459    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 21:39:25.558105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:25.558021    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:39:25.610778 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:25.609964    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"]
Oct 30 21:39:25.610778 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:25.610219    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz" podUID="3a782cd5-ce3f-4709-b343-4a15776d3a76" containerName="maas-api" containerID="cri-o://7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be" gracePeriod=30
Oct 30 21:39:25.926611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:25.926582    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:39:26.032761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.030390    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a782cd5-ce3f-4709-b343-4a15776d3a76-kube-api-access-4g4rm\" (UniqueName: \"kubernetes.io/projected/3a782cd5-ce3f-4709-b343-4a15776d3a76-kube-api-access-4g4rm\") pod \"3a782cd5-ce3f-4709-b343-4a15776d3a76\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") "
Oct 30 21:39:26.032761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.030499    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-maas-api-tls\") pod \"3a782cd5-ce3f-4709-b343-4a15776d3a76\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") "
Oct 30 21:39:26.032761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.030531    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-metrics-certs\") pod \"3a782cd5-ce3f-4709-b343-4a15776d3a76\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") "
Oct 30 21:39:26.032761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.031974    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a782cd5-ce3f-4709-b343-4a15776d3a76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a782cd5-ce3f-4709-b343-4a15776d3a76-openshift-service-ca\") pod \"3a782cd5-ce3f-4709-b343-4a15776d3a76\" (UID: \"3a782cd5-ce3f-4709-b343-4a15776d3a76\") "
Oct 30 21:39:26.033607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.033284    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a782cd5-ce3f-4709-b343-4a15776d3a76-openshift-service-ca" pod "3a782cd5-ce3f-4709-b343-4a15776d3a76" (UID: "3a782cd5-ce3f-4709-b343-4a15776d3a76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:26.037015 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.036729    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a782cd5-ce3f-4709-b343-4a15776d3a76-kube-api-access-4g4rm" pod "3a782cd5-ce3f-4709-b343-4a15776d3a76" (UID: "3a782cd5-ce3f-4709-b343-4a15776d3a76"). InnerVolumeSpecName "kube-api-access-4g4rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:26.054106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.053753    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-maas-api-tls" pod "3a782cd5-ce3f-4709-b343-4a15776d3a76" (UID: "3a782cd5-ce3f-4709-b343-4a15776d3a76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:26.058841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.058777    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-metrics-certs" pod "3a782cd5-ce3f-4709-b343-4a15776d3a76" (UID: "3a782cd5-ce3f-4709-b343-4a15776d3a76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:26.134610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.134512    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a782cd5-ce3f-4709-b343-4a15776d3a76-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:39:26.134610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.134547    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4rm\" (UniqueName: \"kubernetes.io/projected/3a782cd5-ce3f-4709-b343-4a15776d3a76-kube-api-access-4g4rm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:39:26.134610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.134567    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:39:26.134610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.134582    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a782cd5-ce3f-4709-b343-4a15776d3a76-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:39:26.330388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.330355    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:26.330924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.330688    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:26.331846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.331823    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:39:26.714566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.714531    5886 generic.go:356] "Generic (PLEG): container finished" podID="3a782cd5-ce3f-4709-b343-4a15776d3a76" containerID="7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be" exitCode=0
Oct 30 21:39:26.714751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.714576    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"
Oct 30 21:39:26.714751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.714611    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz" event={"ID":"3a782cd5-ce3f-4709-b343-4a15776d3a76","Type":"ContainerDied","Data":"7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be"}
Oct 30 21:39:26.714751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.714662    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz" event={"ID":"3a782cd5-ce3f-4709-b343-4a15776d3a76","Type":"ContainerDied","Data":"f02e3d6352f9fe27214f0ba911124a96278d45c2a32bc17831293c4518110102"}
Oct 30 21:39:26.714751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.714684    5886 scope.go:122] "RemoveContainer" containerID="7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be"
Oct 30 21:39:26.725334 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.725314    5886 scope.go:122] "RemoveContainer" containerID="7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be"
Oct 30 21:39:26.725638 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:39:26.725615    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be\": container with ID starting with 7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be not found: ID does not exist" containerID="7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be"
Oct 30 21:39:26.725708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.725645    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be"} err="failed to get container status \"7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be\": rpc error: code = NotFound desc = could not find container \"7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be\": container with ID starting with 7d8fc49a8c685ed5e2c34c89b63cd27997ac885bc0028d25084bab955fcc72be not found: ID does not exist"
Oct 30 21:39:26.766522 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.766483    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"]
Oct 30 21:39:26.769960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:26.769935    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e4693-74df8668b-wknsz"]
Oct 30 21:39:27.988278 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:27.988241    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a782cd5-ce3f-4709-b343-4a15776d3a76" path="/var/lib/kubelet/pods/3a782cd5-ce3f-4709-b343-4a15776d3a76/volumes"
Oct 30 21:39:32.864249 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.864209    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"]
Oct 30 21:39:32.864791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.864775    5886 cpu_manager.go:414] "containerMap: removing container" podUID="3a782cd5-ce3f-4709-b343-4a15776d3a76" containerName="maas-api"
Oct 30 21:39:32.864791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.864790    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a782cd5-ce3f-4709-b343-4a15776d3a76" containerName="maas-api"
Oct 30 21:39:32.864917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.864828    5886 cpu_manager.go:414] "containerMap: removing container" podUID="5a6c6198-7f2e-4a4f-b180-d44786fa955b" containerName="payload-processing"
Oct 30 21:39:32.864917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.864834    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6c6198-7f2e-4a4f-b180-d44786fa955b" containerName="payload-processing"
Oct 30 21:39:32.865025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.864924    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a782cd5-ce3f-4709-b343-4a15776d3a76" containerName="maas-api"
Oct 30 21:39:32.865025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.864939    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6c6198-7f2e-4a4f-b180-d44786fa955b" containerName="payload-processing"
Oct 30 21:39:32.869952 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.869926    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.897578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.897544    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"]
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916351    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916468    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916536    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916565    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916607    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916633    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916667    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916711    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkjn\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-kube-api-access-lhkjn\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:32.917005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:32.916735    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.017681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.017642    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.017859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.017740    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.017859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.017791    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.017859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.017820    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.017993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.017862    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.017993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.017889    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.017993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.017923    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.017993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.017984    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkjn\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-kube-api-access-lhkjn\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.018226 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.018013    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.018284 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.018247    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.018660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.018483    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.018660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.018502    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.018880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.018789    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.019108 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.019086    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.020719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.020697    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.020827 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.020810    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.036255 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.036233    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.036798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.036777    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkjn\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-kube-api-access-lhkjn\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.188469 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.188434    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:33.526885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.526793    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"]
Oct 30 21:39:33.533009 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:39:33.532979    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5962f18_d7e2_4dc4_83c9_a1f3d784b59e.slice/crio-624da93142e28133bdd20d9f58f97c23f229cecc1d4155def79f855dab10edec WatchSource:0}: Error finding container 624da93142e28133bdd20d9f58f97c23f229cecc1d4155def79f855dab10edec: Status 404 returned error can't find the container with id 624da93142e28133bdd20d9f58f97c23f229cecc1d4155def79f855dab10edec
Oct 30 21:39:33.535748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.535709    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:33.535857 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.535782    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:33.535857 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.535817    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:39:33.790241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.782757    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" event={"ID":"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e","Type":"ContainerStarted","Data":"722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8"}
Oct 30 21:39:33.790241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.782833    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" event={"ID":"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e","Type":"ContainerStarted","Data":"624da93142e28133bdd20d9f58f97c23f229cecc1d4155def79f855dab10edec"}
Oct 30 21:39:33.825796 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:33.825739    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podStartSLOduration=1.82572365 podStartE2EDuration="1.82572365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:33.823006262 +0000 UTC m=+1773.952257720" watchObservedRunningTime="2026-10-30 21:39:33.82572365 +0000 UTC m=+1773.954975107"
Oct 30 21:39:34.189124 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:34.189081    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:34.190761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:34.190733    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:39:34.190903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:34.190826    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 21:39:35.190182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:35.189575    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:39:35.190182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:35.189653    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 21:39:36.192180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:36.191500    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:39:36.192180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:36.191559    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 21:39:37.192589 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:37.192550    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:39:37.193232 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:37.193193    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 21:39:38.192552 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:38.192511    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:39:38.192836 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:38.192807    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 21:39:39.193183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:39.192285    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:39:39.193183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:39.192371    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 21:39:39.434638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:39.434585    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:39.730661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:39.730609    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:39.731297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:39.731265    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:39:39.796298 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:39.796249    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:39.807278 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:39.807235    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.160988 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:40.154251    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.217259 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:40.212348    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:39:40.217259 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:40.212411    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 21:39:40.310891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:40.300487    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.732530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:40.732021    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:40.733398 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:40.733362    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:39:40.788615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:40.788101    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:40.788615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:40.788183    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:41.119769 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:41.119585    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:39:41.125572 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:39:41.125351    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:39:41.190783 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:41.190741    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:39:41.191068 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:41.191037    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 21:39:42.477658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:42.477614    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:42.478961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:42.478937    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:42.480195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:42.480125    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 21:39:54.671963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.671924    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"]
Oct 30 21:39:54.676362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.676338    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.693038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.693016    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-7a4de0-serving-cert\""
Oct 30 21:39:54.708064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.708033    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"]
Oct 30 21:39:54.720047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.720012    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.720246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.720058    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw742\" (UniqueName: \"kubernetes.io/projected/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-kube-api-access-pw742\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.720306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.720244    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-metrics-certs\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.720555 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.720417    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.821972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.821938    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-metrics-certs\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.822144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.822047    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.822144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.822107    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.822144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.822123    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw742\" (UniqueName: \"kubernetes.io/projected/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-kube-api-access-pw742\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.822788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.822756    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.824502 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.824477    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-metrics-certs\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.824590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.824500    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-maas-api-tls\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.840520 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.840494    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw742\" (UniqueName: \"kubernetes.io/projected/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-kube-api-access-pw742\") pod \"maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:54.993428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:54.993322    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:55.298979 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:55.298884    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"]
Oct 30 21:39:55.491820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:55.491781    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29" event={"ID":"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55","Type":"ContainerStarted","Data":"769722d45fc6419c60bc29115da0ac30c66c0f6ed7f822546861d3791adfa9ff"}
Oct 30 21:39:56.505180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:56.504534    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29" event={"ID":"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55","Type":"ContainerStarted","Data":"bf88b63ff3e30e9638d720b50eaf8a8447116c6a61c38f882b6aabdb4af9c858"}
Oct 30 21:39:56.505180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:56.504625    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:39:56.546599 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:39:56.546550    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29" podStartSLOduration=2.212782864 podStartE2EDuration="2.546536501s" totalImagesPullingTime="333.753637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:56.546477155 +0000 UTC m=+1796.675728625" watchObservedRunningTime="2026-10-30 21:39:56.546536501 +0000 UTC m=+1796.675787958"
Oct 30 21:40:02.542564 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:02.542486    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:40:02.979359 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:02.979324    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"]
Oct 30 21:40:02.984849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:02.984823    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.033676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.022243    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"]
Oct 30 21:40:03.148768 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.148733    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.148963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.148803    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.148963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.148899    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.148963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.148955    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.149089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.148990    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.149137 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.149106    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.149210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.149139    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv45g\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-kube-api-access-bv45g\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.149256 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.149233    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.149299 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.149285    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.249965    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250023    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250071    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250098    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv45g\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-kube-api-access-bv45g\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250512 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250179    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250512 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250212    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250512 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250409    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-data\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250512 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250453    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-credential-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250725 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250698    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250794    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.250832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.250822    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.251056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.251037    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-certs\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.251202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.251179    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-socket\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.251487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.251469    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istiod-ca-cert\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.252472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.252454    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-envoy\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.252681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.252664    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-podinfo\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.271752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.271719    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv45g\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-kube-api-access-bv45g\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.272023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.272004    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-token\") pod \"e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") " pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.305581 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.305537    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:03.626492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.626411    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"]
Oct 30 21:40:03.629654 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:40:03.629629    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4109ea5_70d6_484d_b8a2_c78c1a57c889.slice/crio-c3674504a4d634bf94d84465dc3cb14c988d941984dd1469a416a4677faa7b6c WatchSource:0}: Error finding container c3674504a4d634bf94d84465dc3cb14c988d941984dd1469a416a4677faa7b6c: Status 404 returned error can't find the container with id c3674504a4d634bf94d84465dc3cb14c988d941984dd1469a416a4677faa7b6c
Oct 30 21:40:03.632721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.632670    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:03.632864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.632838    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:03.632958 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:03.632935    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:04.599259 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:04.599211    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" event={"ID":"b4109ea5-70d6-484d-b8a2-c78c1a57c889","Type":"ContainerStarted","Data":"f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182"}
Oct 30 21:40:04.599481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:04.599279    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" event={"ID":"b4109ea5-70d6-484d-b8a2-c78c1a57c889","Type":"ContainerStarted","Data":"c3674504a4d634bf94d84465dc3cb14c988d941984dd1469a416a4677faa7b6c"}
Oct 30 21:40:04.653771 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:04.653706    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podStartSLOduration=2.653691108 podStartE2EDuration="2.653691108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:04.651040517 +0000 UTC m=+1804.780291975" watchObservedRunningTime="2026-10-30 21:40:04.653691108 +0000 UTC m=+1804.782942564"
Oct 30 21:40:05.305751 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:05.305717    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:05.307445 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:05.307416    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 21:40:05.307584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:05.307475    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 21:40:06.306345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:06.306310    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 21:40:06.306851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:06.306367    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 21:40:07.306308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:07.306269    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 21:40:07.306525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:07.306334    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 21:40:08.306737 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:08.306698    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 21:40:08.307173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:08.306754    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 21:40:09.306482 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:09.306438    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 21:40:09.306671 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:09.306502    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 21:40:10.306954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:10.306922    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 21:40:10.307457 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:10.306982    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 21:40:11.028832 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:40:11.028733    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:40:11.311579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:11.311499    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:11.312025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:11.312003    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:11.312936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:11.312913    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 21:40:11.993825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:11.993791    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"]
Oct 30 21:40:11.999177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:11.999143    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.020193 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.020151    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"]
Oct 30 21:40:12.027832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.027798    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ea40ee-openshift-default-dockercfg-8mf6h\""
Oct 30 21:40:12.109404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109350    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1254d30-70fd-4247-b20b-18233b6a6992-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.109609 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109427    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.109609 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109582    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.109731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109666    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.109794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109732    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srlkf\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-kube-api-access-srlkf\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.109794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109785    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.109895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109834    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.109895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109887    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.109996 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.109915    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1254d30-70fd-4247-b20b-18233b6a6992-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212194    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212332    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212388    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srlkf\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-kube-api-access-srlkf\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212450    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212505    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212560    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212585    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1254d30-70fd-4247-b20b-18233b6a6992-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212664    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1254d30-70fd-4247-b20b-18233b6a6992-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.212695    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.225178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.221615    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.227714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.227671    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.228586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.228566    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.230080 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.230062    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1254d30-70fd-4247-b20b-18233b6a6992-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.230308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.230270    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.243757 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.243721    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1254d30-70fd-4247-b20b-18233b6a6992-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.260223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.253035    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.260223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.256806    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srlkf\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-kube-api-access-srlkf\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.260223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.257354    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.313987 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.313941    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:12.677148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.677125    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"]
Oct 30 21:40:12.680451 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:40:12.680413    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1254d30_70fd_4247_b20b_18233b6a6992.slice/crio-159037368e78107613e8561936c59a9317e0a14e503256ffd8df9cc6a7e63092 WatchSource:0}: Error finding container 159037368e78107613e8561936c59a9317e0a14e503256ffd8df9cc6a7e63092: Status 404 returned error can't find the container with id 159037368e78107613e8561936c59a9317e0a14e503256ffd8df9cc6a7e63092
Oct 30 21:40:12.683621 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.683574    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:12.683755 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.683670    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:12.683755 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:12.683715    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:13.716188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:13.708149    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" event={"ID":"c1254d30-70fd-4247-b20b-18233b6a6992","Type":"ContainerStarted","Data":"df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8"}
Oct 30 21:40:13.716188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:13.708241    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" event={"ID":"c1254d30-70fd-4247-b20b-18233b6a6992","Type":"ContainerStarted","Data":"159037368e78107613e8561936c59a9317e0a14e503256ffd8df9cc6a7e63092"}
Oct 30 21:40:13.805181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:13.801707    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podStartSLOduration=2.801688843 podStartE2EDuration="2.801688843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:13.78664359 +0000 UTC m=+1813.915895048" watchObservedRunningTime="2026-10-30 21:40:13.801688843 +0000 UTC m=+1813.930940303"
Oct 30 21:40:14.317180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:14.316407    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:14.317180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:14.316580    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:14.317180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:14.316623    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:14.930280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:14.930233    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:15.154182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.151231    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:40:15.265413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.265294    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:15.394659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.308241    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:15.394659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.325717    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:15.394659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.325781    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:15.714924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.714883    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:15.723630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.723598    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"]
Oct 30 21:40:15.804213 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.803132    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.832183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.829395    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.928582    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.928702    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.928742    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.928772    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076366e0-68f8-408b-9867-a5ef93ef1987-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.928808    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpptx\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-kube-api-access-cpptx\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.928839    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.928961    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.929040    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076366e0-68f8-408b-9867-a5ef93ef1987-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.929180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.929093    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:15.968237 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:15.942227    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"]
Oct 30 21:40:16.139343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.127293    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.197182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.169843    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.409240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.384362    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.409240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.200034    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.488273 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.199556    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.496319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.496278    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.541319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.531319    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076366e0-68f8-408b-9867-a5ef93ef1987-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.541319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.531401    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpptx\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-kube-api-access-cpptx\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.541319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.531561    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.541319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.531702    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.541319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.531798    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076366e0-68f8-408b-9867-a5ef93ef1987-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.541319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.532056    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.541319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.532209    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.541319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.533565    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-data\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.592183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.566869    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-credential-socket\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.629180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.603983    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076366e0-68f8-408b-9867-a5ef93ef1987-istiod-ca-cert\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.661273 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.628198    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-certs\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.661273 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.657465    5886 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.661273 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.657604    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 21:40:16.778182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.762972    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 21:40:16.806204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.799678    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpptx\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-kube-api-access-cpptx\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.806613 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.806585    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076366e0-68f8-408b-9867-a5ef93ef1987-istio-podinfo\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:16.848991 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.844342    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:16.876714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.874764    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:16.876714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:16.874847    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:17.011231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.007348    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-envoy\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:17.325194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.324262    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:17.328352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.328223    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:17.329262 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:40:17.329220    5886 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.129s"
Oct 30 21:40:17.329385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.329300    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:40:17.332098 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.331930    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:17.383966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.372580    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:17.387905 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.375370    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:40:17.405391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.400796    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:40:17.425199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.425121    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:40:17.425199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.425202    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:40:17.425548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.380480    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-istio-token\") pod \"e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") " pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:17.425865 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.389064    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:17.429629 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.429606    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 21:40:17.648378 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.648349    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c"
Oct 30 21:40:17.666655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.666626    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 21:40:17.669738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.669256    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:17.670027 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.669962    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:40:17.685341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.676519    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 21:40:17.737076 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:17.737037    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:18.352213 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.352104    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:40:18.353554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.353524    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:18.353750 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.353724    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:18.443343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.427681    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:40:18.622274 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.622187    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 21:40:18.715770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.707053    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"]
Oct 30 21:40:18.720721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.720679    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:18.720900 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.720879    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:18.720999 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:18.720986    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:19.314952 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:19.314917    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:19.315219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:19.315188    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:19.502473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:19.497551    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" event={"ID":"076366e0-68f8-408b-9867-a5ef93ef1987","Type":"ContainerStarted","Data":"cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162"}
Oct 30 21:40:19.502473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:19.497620    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" event={"ID":"076366e0-68f8-408b-9867-a5ef93ef1987","Type":"ContainerStarted","Data":"34ddc3c23e6a6032ebd4a3227639fc6eb0a1d8a3a58cc3ece69284ec9a86ba36"}
Oct 30 21:40:19.539971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:19.539907    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podStartSLOduration=4.539891408 podStartE2EDuration="4.539891408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:19.538582383 +0000 UTC m=+1819.667833839" watchObservedRunningTime="2026-10-30 21:40:19.539891408 +0000 UTC m=+1819.669142866"
Oct 30 21:40:19.742184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:19.738626    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:19.742184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:19.740384    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:40:19.742184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:19.740464    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 21:40:20.318352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:20.318302    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:20.318539 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:20.318369    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:20.737984 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:20.737948    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:40:20.738413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:20.738002    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 21:40:21.315144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:21.315109    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:21.315355 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:21.315188    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:21.737794 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:21.737753    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:40:21.737998 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:21.737823    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 21:40:22.314765 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:22.314724    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:40:22.315326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:22.314792    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 30 21:40:22.737773 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:22.737730    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:40:22.737945 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:22.737799    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 21:40:23.738362 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:23.738328    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:40:23.738737 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:23.738384    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 21:40:24.356309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:24.356273    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:40:24.356501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:24.356331    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:24.775369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:24.775331    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:40:24.786506 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:24.786468    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 21:40:24.893192 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:24.890457    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:24.893760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:24.893733    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:24.920660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:24.920587    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 21:40:25.737892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:25.737852    5886 patch_prober.go:29] interesting pod/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:40:25.738660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:25.737921    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Oct 30 21:40:26.742564 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:26.742534    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:26.743012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:26.742998    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:26.743750 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:26.743727    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:40:27.070018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.069983    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"]
Oct 30 21:40:27.074293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.074270    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.095698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.095666    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"]
Oct 30 21:40:27.134258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134220    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.134461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134286    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.134461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134361    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vb4b\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-kube-api-access-9vb4b\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.134461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134410    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d73009a-9704-4c93-80fa-df3da2f71609-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.134461 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134433    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.134785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134585    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.134785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134629    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d73009a-9704-4c93-80fa-df3da2f71609-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.134785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134693    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.134785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.134728    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236101    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236200    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d73009a-9704-4c93-80fa-df3da2f71609-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236250    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236283    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236334    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236378    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236420    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vb4b\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-kube-api-access-9vb4b\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236491    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d73009a-9704-4c93-80fa-df3da2f71609-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236529    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236577    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-certs\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236861 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236655    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-data\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236980 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236951    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-credential-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.236980 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.236972    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-socket\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.237333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.237313    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d73009a-9704-4c93-80fa-df3da2f71609-istiod-ca-cert\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.238636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.238615    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-envoy\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.238856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.238838    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d73009a-9704-4c93-80fa-df3da2f71609-istio-podinfo\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.256267 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.256238    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-istio-token\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.256508 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.256488    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vb4b\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-kube-api-access-9vb4b\") pod \"e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") " pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.389864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.389759    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:27.718082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.718053    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"]
Oct 30 21:40:27.721851 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:40:27.721817    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d73009a_9704_4c93_80fa_df3da2f71609.slice/crio-cee59950025089428ffd85e254b6e798d1ccfea7d943a43371ed8190ae4b4ef9 WatchSource:0}: Error finding container cee59950025089428ffd85e254b6e798d1ccfea7d943a43371ed8190ae4b4ef9: Status 404 returned error can't find the container with id cee59950025089428ffd85e254b6e798d1ccfea7d943a43371ed8190ae4b4ef9
Oct 30 21:40:27.725000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.724951    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:27.725084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.725048    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:27.725144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:27.725096    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:40:28.942297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:28.942256    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" event={"ID":"1d73009a-9704-4c93-80fa-df3da2f71609","Type":"ContainerStarted","Data":"e6b0131ad01eb682c5914eefdea2e728432a5ad7c42ffc8451e7c8ae5821678f"}
Oct 30 21:40:28.942932 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:28.942903    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" event={"ID":"1d73009a-9704-4c93-80fa-df3da2f71609","Type":"ContainerStarted","Data":"cee59950025089428ffd85e254b6e798d1ccfea7d943a43371ed8190ae4b4ef9"}
Oct 30 21:40:29.095148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:29.095068    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podStartSLOduration=2.095050078 podStartE2EDuration="2.095050078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:29.094511621 +0000 UTC m=+1829.223763079" watchObservedRunningTime="2026-10-30 21:40:29.095050078 +0000 UTC m=+1829.224301529"
Oct 30 21:40:29.400343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:29.400300    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:29.408365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:29.408317    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:40:29.408686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:29.408638    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:40:30.370185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:30.369263    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:30.393326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:30.393285    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:40:30.393915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:30.393879    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:40:30.444184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:30.435309    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Oct 30 21:40:30.695180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:30.694490    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:40:30.752257 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:30.695812    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c" podUID="24915d9d-898e-4455-8c32-a4ff2bac9251" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:30.752257 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:30.709241    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:31.442937 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:31.441280    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:40:31.442937 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:31.441341    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:40:31.472902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:31.468251    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:31.472902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:31.469424    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:40:32.045135 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:32.044969    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 21:40:32.390610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:32.390519    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:40:32.390985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:32.390951    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:40:33.392696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:33.392648    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:40:33.393267 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:33.392734    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:40:34.400056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:34.399667    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:40:34.400056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:34.399743    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:40:35.391136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:35.391095    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:40:35.391387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:35.391219    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:40:37.434328 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:37.434274    5886 patch_prober.go:29] interesting pod/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:40:37.434855 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:37.434367    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:37.677428 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:37.677391    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:37.679600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:37.679571    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:37.696427 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:37.696346    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:40:41.031654 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:40:41.031611    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:40:42.211994 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.211947    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"]
Oct 30 21:40:42.223421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.223390    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:42.230279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.230244    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"]
Oct 30 21:40:42.239579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.239540    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e06b90\""
Oct 30 21:40:42.253121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.253091    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e06b90-dockercfg-9lpzx\""
Oct 30 21:40:42.291221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.291187    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtzz\" (UniqueName: \"kubernetes.io/projected/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-kube-api-access-2qtzz\") pod \"payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc\" (UID: \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:42.291385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.291251    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc\" (UID: \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:42.392082 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.392049    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtzz\" (UniqueName: \"kubernetes.io/projected/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-kube-api-access-2qtzz\") pod \"payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc\" (UID: \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:42.392296 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.392209    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc\" (UID: \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:42.392910 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.392889    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc\" (UID: \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:42.410001 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.409973    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtzz\" (UniqueName: \"kubernetes.io/projected/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-kube-api-access-2qtzz\") pod \"payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc\" (UID: \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:42.562622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.562547    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:42.862722 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:42.862637    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"]
Oct 30 21:40:42.866556 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:40:42.866519    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6fb98f_3e4b_4612_a02c_5f554ff5b7f2.slice/crio-90165df685e47dc96c4e43b8211caf55fc4b5c6dacbeb48aaca036e6aedff7a8 WatchSource:0}: Error finding container 90165df685e47dc96c4e43b8211caf55fc4b5c6dacbeb48aaca036e6aedff7a8: Status 404 returned error can't find the container with id 90165df685e47dc96c4e43b8211caf55fc4b5c6dacbeb48aaca036e6aedff7a8
Oct 30 21:40:43.271020 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:43.270981    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc" event={"ID":"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2","Type":"ContainerStarted","Data":"4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a"}
Oct 30 21:40:43.271599 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:43.271040    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc" event={"ID":"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2","Type":"ContainerStarted","Data":"90165df685e47dc96c4e43b8211caf55fc4b5c6dacbeb48aaca036e6aedff7a8"}
Oct 30 21:40:43.271599 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:43.271065    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:40:43.317895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:43.317828    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc" podStartSLOduration=1.317813243 podStartE2EDuration="1.317813243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:43.313946173 +0000 UTC m=+1843.443197647" watchObservedRunningTime="2026-10-30 21:40:43.317813243 +0000 UTC m=+1843.447064700"
Oct 30 21:40:43.929740 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:43.929706    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"]
Oct 30 21:40:43.934069 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:43.934045    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:43.951114 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:43.951087    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"]
Oct 30 21:40:43.958989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:43.958871    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-5494e8-serving-cert\""
Oct 30 21:40:44.012841 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.012810    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj5zf\" (UniqueName: \"kubernetes.io/projected/2811cac3-c5ae-4ae5-988b-c4839e9c8364-kube-api-access-cj5zf\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.013008 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.012948    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.013008 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.012999    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2811cac3-c5ae-4ae5-988b-c4839e9c8364-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.013130 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.013117    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.114764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.114717    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj5zf\" (UniqueName: \"kubernetes.io/projected/2811cac3-c5ae-4ae5-988b-c4839e9c8364-kube-api-access-cj5zf\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.114967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.114877    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.114967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.114945    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2811cac3-c5ae-4ae5-988b-c4839e9c8364-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.115110 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.115045    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.115200 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:40:44.115178    5886 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-serving-cert: secret "maas-api-e2e-shared-a-w4-5494e8-serving-cert" not found
Oct 30 21:40:44.115263 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:40:44.115240    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls podName:2811cac3-c5ae-4ae5-988b-c4839e9c8364 nodeName:}" failed. No retries permitted until 2026-10-30 21:40:44.615225255 +0000 UTC m=+1844.744476694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls") pod "maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw" (UID: "2811cac3-c5ae-4ae5-988b-c4839e9c8364") : secret "maas-api-e2e-shared-a-w4-5494e8-serving-cert" not found
Oct 30 21:40:44.115594 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.115574    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2811cac3-c5ae-4ae5-988b-c4839e9c8364-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.117748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.117726    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.140317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.140272    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj5zf\" (UniqueName: \"kubernetes.io/projected/2811cac3-c5ae-4ae5-988b-c4839e9c8364-kube-api-access-cj5zf\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.625835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.625800    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.628214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.628191    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:44.847711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:44.847679    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:45.152908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:45.152732    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"]
Oct 30 21:40:45.155804 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:40:45.155743    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2811cac3_c5ae_4ae5_988b_c4839e9c8364.slice/crio-b9cade5b99db41b0ea1d33d47cfbe6cb950e6c8d48a6e80ca555809c8a399751 WatchSource:0}: Error finding container b9cade5b99db41b0ea1d33d47cfbe6cb950e6c8d48a6e80ca555809c8a399751: Status 404 returned error can't find the container with id b9cade5b99db41b0ea1d33d47cfbe6cb950e6c8d48a6e80ca555809c8a399751
Oct 30 21:40:45.322879 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:45.322835    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw" event={"ID":"2811cac3-c5ae-4ae5-988b-c4839e9c8364","Type":"ContainerStarted","Data":"b9cade5b99db41b0ea1d33d47cfbe6cb950e6c8d48a6e80ca555809c8a399751"}
Oct 30 21:40:46.331459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:46.331416    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw" event={"ID":"2811cac3-c5ae-4ae5-988b-c4839e9c8364","Type":"ContainerStarted","Data":"51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8"}
Oct 30 21:40:46.331882 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:46.331490    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:46.371558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:46.371504    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw" podStartSLOduration=3.044728216 podStartE2EDuration="3.371488947s" totalImagesPullingTime="326.760731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:46.36878254 +0000 UTC m=+1846.498033998" watchObservedRunningTime="2026-10-30 21:40:46.371488947 +0000 UTC m=+1846.500740404"
Oct 30 21:40:52.344504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:52.344475    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 21:40:54.319181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:40:54.304327    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 21:41:00.621705 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.621667    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"]
Oct 30 21:41:00.625885 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.625866    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.645343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.645315    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"]
Oct 30 21:41:00.672817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.672783    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e06b90-serving-cert\""
Oct 30 21:41:00.760863 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.760828    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.761068 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.760876    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.761139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.761089    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fml9t\" (UniqueName: \"kubernetes.io/projected/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-kube-api-access-fml9t\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.761246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.761135    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.871002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.870951    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.871220 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.871021    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.871303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.871271    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fml9t\" (UniqueName: \"kubernetes.io/projected/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-kube-api-access-fml9t\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.871363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.871341    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.872005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.871939    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.887703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.879971    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.900828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.899788    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.902688 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.901816    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fml9t\" (UniqueName: \"kubernetes.io/projected/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-kube-api-access-fml9t\") pod \"maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:00.939915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:00.939883    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:01.257198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:01.257171    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"]
Oct 30 21:41:01.259974 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:41:01.259943    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed962bd_2ed6_44b4_a4e3_bef92fc054b8.slice/crio-a82b0addefc1ed6bb48742a639ef799c7b584566d7a3598b809b85df87fb304b WatchSource:0}: Error finding container a82b0addefc1ed6bb48742a639ef799c7b584566d7a3598b809b85df87fb304b: Status 404 returned error can't find the container with id a82b0addefc1ed6bb48742a639ef799c7b584566d7a3598b809b85df87fb304b
Oct 30 21:41:01.475042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:01.475005    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66" event={"ID":"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8","Type":"ContainerStarted","Data":"a82b0addefc1ed6bb48742a639ef799c7b584566d7a3598b809b85df87fb304b"}
Oct 30 21:41:02.514180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:02.512714    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66" event={"ID":"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8","Type":"ContainerStarted","Data":"c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680"}
Oct 30 21:41:02.529195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:02.526472    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:08.095780 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.095719    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66" podStartSLOduration=7.705679962 podStartE2EDuration="8.095698567s" totalImagesPullingTime="390.018605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:02.579481114 +0000 UTC m=+1862.708732608" watchObservedRunningTime="2026-10-30 21:41:08.095698567 +0000 UTC m=+1868.224950030"
Oct 30 21:41:08.096817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.096791    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"]
Oct 30 21:41:08.102512 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.102488    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.129379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.129336    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"]
Oct 30 21:41:08.150886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.150845    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr96c\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-kube-api-access-dr96c\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.151071 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.150973    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.151071 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.151027    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.151071 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.151065    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.151279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.151210    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.151336 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.151284    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.151396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.151348    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.151459 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.151419    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.151578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.151551    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.194280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.194239    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-09eabe-openshift-default-dockercfg-dqwhz\""
Oct 30 21:41:08.252947 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.252913    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.252981    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr96c\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-kube-api-access-dr96c\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253017    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253041    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253060    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253085    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253469 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253210    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253469 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253242    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253469 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253303    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253529    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-certs\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253640 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253598    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-credential-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253732 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253644    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-socket\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253732 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253711    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-data\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.253873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.253852    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istiod-ca-cert\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.255466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.255445    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-envoy\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.255735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.255713    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-podinfo\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.271603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.271572    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-token\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.271726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.271695    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr96c\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-kube-api-access-dr96c\") pod \"e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") " pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.421510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.421476    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:08.530891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.530857    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 21:41:08.759097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.759060    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"]
Oct 30 21:41:08.766634 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.766590    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:08.766877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.766848    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:08.767049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:08.766903    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:09.610422 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:09.610373    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" event={"ID":"970e4f4a-4008-4139-a720-0c6d8e1bf0b7","Type":"ContainerStarted","Data":"de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6"}
Oct 30 21:41:09.610933 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:09.610479    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" event={"ID":"970e4f4a-4008-4139-a720-0c6d8e1bf0b7","Type":"ContainerStarted","Data":"d8e5b7c6b6a64fc020683a1c5391e0a047ef741045b69260a125f3dd06a32a3a"}
Oct 30 21:41:09.682176 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:09.678277    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" podStartSLOduration=1.6782562890000001 podStartE2EDuration="1.678256289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:09.667687816 +0000 UTC m=+1869.796939274" watchObservedRunningTime="2026-10-30 21:41:09.678256289 +0000 UTC m=+1869.807507748"
Oct 30 21:41:10.421760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:10.421722    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:10.423069 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:10.423043    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 21:41:10.423139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:10.423096    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 21:41:11.029948 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:41:11.029893    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:41:11.422661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:11.422625    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 21:41:11.422847 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:11.422702    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 21:41:12.422850 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:12.422805    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 21:41:12.423436 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:12.422890    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 21:41:13.422317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:13.422278    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 21:41:13.422527 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:13.422356    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 21:41:14.315221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.315186    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"]
Oct 30 21:41:14.320068 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.320042    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:14.337495 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.337463    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-f49db2\""
Oct 30 21:41:14.348964 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.348938    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-f49db2-dockercfg-rrpql\""
Oct 30 21:41:14.355034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.355006    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"]
Oct 30 21:41:14.396183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.374243    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75bv\" (UniqueName: \"kubernetes.io/projected/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-kube-api-access-x75bv\") pod \"payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf\" (UID: \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:14.397540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.397508    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf\" (UID: \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:14.532177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.530684    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75bv\" (UniqueName: \"kubernetes.io/projected/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-kube-api-access-x75bv\") pod \"payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf\" (UID: \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:14.532177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.530754    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf\" (UID: \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:14.532177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.531617    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf\" (UID: \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:14.630252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.621423    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 30 21:41:14.678300 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.678253    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 30 21:41:14.744265 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.744229    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75bv\" (UniqueName: \"kubernetes.io/projected/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-kube-api-access-x75bv\") pod \"payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf\" (UID: \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:14.934849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:14.934807    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:15.280917 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:15.280852    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"]
Oct 30 21:41:15.284318 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:41:15.284274    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbb8d0d_a8cb_4b10_9ff0_3b3d32075037.slice/crio-f58f5f402fa74a79e5684d7299905464d8efecf50a40a74ed453906986932b35 WatchSource:0}: Error finding container f58f5f402fa74a79e5684d7299905464d8efecf50a40a74ed453906986932b35: Status 404 returned error can't find the container with id f58f5f402fa74a79e5684d7299905464d8efecf50a40a74ed453906986932b35
Oct 30 21:41:15.442081 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:15.441982    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:15.442892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:15.442790    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:15.444080 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:15.444055    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 21:41:15.723743 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:15.723698    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf" event={"ID":"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037","Type":"ContainerStarted","Data":"64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de"}
Oct 30 21:41:15.723951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:15.723922    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:15.724021 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:15.723960    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf" event={"ID":"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037","Type":"ContainerStarted","Data":"f58f5f402fa74a79e5684d7299905464d8efecf50a40a74ed453906986932b35"}
Oct 30 21:41:15.763103 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:15.763044    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf" podStartSLOduration=1.763029366 podStartE2EDuration="1.763029366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:15.761499654 +0000 UTC m=+1875.890751110" watchObservedRunningTime="2026-10-30 21:41:15.763029366 +0000 UTC m=+1875.892280822"
Oct 30 21:41:20.476747 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.476710    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"]
Oct 30 21:41:20.488388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.488358    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"]
Oct 30 21:41:20.488561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.488479    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:20.575219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.573239    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2lz\" (UniqueName: \"kubernetes.io/projected/e9cef9fb-a937-4586-ad66-b62771a6d36a-kube-api-access-js2lz\") pod \"payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh\" (UID: \"e9cef9fb-a937-4586-ad66-b62771a6d36a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:20.575219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.573352    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cef9fb-a937-4586-ad66-b62771a6d36a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh\" (UID: \"e9cef9fb-a937-4586-ad66-b62771a6d36a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:20.575219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.575094    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-ae6492\""
Oct 30 21:41:20.575526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.575241    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-ae6492-dockercfg-gvgmh\""
Oct 30 21:41:20.675577 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.675489    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cef9fb-a937-4586-ad66-b62771a6d36a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh\" (UID: \"e9cef9fb-a937-4586-ad66-b62771a6d36a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:20.675984 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.675958    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2lz\" (UniqueName: \"kubernetes.io/projected/e9cef9fb-a937-4586-ad66-b62771a6d36a-kube-api-access-js2lz\") pod \"payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh\" (UID: \"e9cef9fb-a937-4586-ad66-b62771a6d36a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:20.677806 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.676366    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cef9fb-a937-4586-ad66-b62771a6d36a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh\" (UID: \"e9cef9fb-a937-4586-ad66-b62771a6d36a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:20.699345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.699309    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2lz\" (UniqueName: \"kubernetes.io/projected/e9cef9fb-a937-4586-ad66-b62771a6d36a-kube-api-access-js2lz\") pod \"payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh\" (UID: \"e9cef9fb-a937-4586-ad66-b62771a6d36a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:20.846996 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:20.846903    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:21.153523 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:21.153438    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"]
Oct 30 21:41:21.161252 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:41:21.161217    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cef9fb_a937_4586_ad66_b62771a6d36a.slice/crio-4fc7e7367d68e060c3b6bd069dba9dff36ecead49acfea51cfd573991c1ae78d WatchSource:0}: Error finding container 4fc7e7367d68e060c3b6bd069dba9dff36ecead49acfea51cfd573991c1ae78d: Status 404 returned error can't find the container with id 4fc7e7367d68e060c3b6bd069dba9dff36ecead49acfea51cfd573991c1ae78d
Oct 30 21:41:21.772995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:21.772958    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh" event={"ID":"e9cef9fb-a937-4586-ad66-b62771a6d36a","Type":"ContainerStarted","Data":"f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6"}
Oct 30 21:41:21.773425 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:21.773021    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh" event={"ID":"e9cef9fb-a937-4586-ad66-b62771a6d36a","Type":"ContainerStarted","Data":"4fc7e7367d68e060c3b6bd069dba9dff36ecead49acfea51cfd573991c1ae78d"}
Oct 30 21:41:21.773425 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:21.773097    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:21.810215 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:21.810143    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh" podStartSLOduration=1.810129466 podStartE2EDuration="1.810129466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:21.808977319 +0000 UTC m=+1881.938228776" watchObservedRunningTime="2026-10-30 21:41:21.810129466 +0000 UTC m=+1881.939380923"
Oct 30 21:41:26.736285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:26.736252    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 21:41:31.720505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.720471    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"]
Oct 30 21:41:31.725554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.725529    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.740115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.740088    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-64d142-openshift-default-dockercfg-cblst\""
Oct 30 21:41:31.755832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.755806    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"]
Oct 30 21:41:31.766913 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.766878    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-certs\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.767258 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.767231    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-token\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.767440 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.767412    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-podinfo\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.767564 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.767544    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-envoy\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.767635 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.767612    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-data\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.767699 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.767672    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-credential-socket\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.767763 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.767703    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istiod-ca-cert\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.767821 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.767786    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtk2t\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-kube-api-access-rtk2t\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.767878 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.767865    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-socket\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.883185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.876805    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-credential-socket\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.924603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.924562    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istiod-ca-cert\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.925818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.925792    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtk2t\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-kube-api-access-rtk2t\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.930733 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.880826    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-credential-socket\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.930733 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.925663    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istiod-ca-cert\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.931616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.931570    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-socket\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.931766 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.927447    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-socket\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.931925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.931908    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-certs\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.932430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.932412    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-token\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.932894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.932875    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-podinfo\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.933561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.933542    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-envoy\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.934887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.934869    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-data\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.935405 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.935243    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-data\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:31.935530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.932290    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-certs\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:32.015279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.990368    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-podinfo\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:32.015279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.995521    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-envoy\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:32.015279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:31.996966    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtk2t\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-kube-api-access-rtk2t\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:32.026938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.026905    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-token\") pod \"e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") " pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:32.042202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.042134    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:32.397404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.397240    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"]
Oct 30 21:41:32.400550 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:41:32.400508    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16257b19_c7b1_4dfc_bcef_e21e9d8c63f7.slice/crio-de84bf8f694aa122b9b377c2fbb3c7d844780622b73c88cf507c67bbe26d8b94 WatchSource:0}: Error finding container de84bf8f694aa122b9b377c2fbb3c7d844780622b73c88cf507c67bbe26d8b94: Status 404 returned error can't find the container with id de84bf8f694aa122b9b377c2fbb3c7d844780622b73c88cf507c67bbe26d8b94
Oct 30 21:41:32.403788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.403751    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:32.403874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.403839    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:32.403914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.403880    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:41:32.785326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.785293    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 21:41:32.932775 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.932713    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" event={"ID":"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7","Type":"ContainerStarted","Data":"c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4"}
Oct 30 21:41:32.933068 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.933032    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" event={"ID":"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7","Type":"ContainerStarted","Data":"de84bf8f694aa122b9b377c2fbb3c7d844780622b73c88cf507c67bbe26d8b94"}
Oct 30 21:41:32.977404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:32.977345    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" podStartSLOduration=1.97732997 podStartE2EDuration="1.97732997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:32.973645095 +0000 UTC m=+1893.102896554" watchObservedRunningTime="2026-10-30 21:41:32.97732997 +0000 UTC m=+1893.106581426"
Oct 30 21:41:33.042361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:33.042274    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:33.049016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:33.048985    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:33.939948 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:33.939922    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:33.940799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:33.940776    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:41:37.304333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.304291    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"]
Oct 30 21:41:37.310920 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.310889    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.355355 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.355316    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"]
Oct 30 21:41:37.364430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.364401    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-f49db2-serving-cert\""
Oct 30 21:41:37.458577 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.458540    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.458748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.458696    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3d315-e31c-4c1f-a441-465b68dcd02f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.458858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.458842    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxnlm\" (UniqueName: \"kubernetes.io/projected/9dd3d315-e31c-4c1f-a441-465b68dcd02f-kube-api-access-dxnlm\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.458907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.458892    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.568231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.566907    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxnlm\" (UniqueName: \"kubernetes.io/projected/9dd3d315-e31c-4c1f-a441-465b68dcd02f-kube-api-access-dxnlm\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.568231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.567276    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.568231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.567880    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.568231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.568018    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3d315-e31c-4c1f-a441-465b68dcd02f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.569812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.568899    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3d315-e31c-4c1f-a441-465b68dcd02f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.590868 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.590809    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxnlm\" (UniqueName: \"kubernetes.io/projected/9dd3d315-e31c-4c1f-a441-465b68dcd02f-kube-api-access-dxnlm\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.591363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.591338    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.593986 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.593957    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:37.634764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:37.634485    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:38.006693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:38.006664    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"]
Oct 30 21:41:38.009833 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:41:38.009803    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd3d315_e31c_4c1f_a441_465b68dcd02f.slice/crio-1ef05c708f9b686b120d144c8821a148604b51414c7be773ec5de84c0f9c1dd4 WatchSource:0}: Error finding container 1ef05c708f9b686b120d144c8821a148604b51414c7be773ec5de84c0f9c1dd4: Status 404 returned error can't find the container with id 1ef05c708f9b686b120d144c8821a148604b51414c7be773ec5de84c0f9c1dd4
Oct 30 21:41:38.979977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:38.979939    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4" event={"ID":"9dd3d315-e31c-4c1f-a441-465b68dcd02f","Type":"ContainerStarted","Data":"df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c"}
Oct 30 21:41:38.980388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:38.979997    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4" event={"ID":"9dd3d315-e31c-4c1f-a441-465b68dcd02f","Type":"ContainerStarted","Data":"1ef05c708f9b686b120d144c8821a148604b51414c7be773ec5de84c0f9c1dd4"}
Oct 30 21:41:38.980388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:38.980137    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:41:39.017256 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:39.017196    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4" podStartSLOduration=1.5848098400000001 podStartE2EDuration="2.017151352s" totalImagesPullingTime="432.341512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:39.01670049 +0000 UTC m=+1899.145951948" watchObservedRunningTime="2026-10-30 21:41:39.017151352 +0000 UTC m=+1899.146402810"
Oct 30 21:41:41.029936 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:41:41.029855    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:41:45.018638 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:41:45.018604    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 21:42:01.523348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.523312    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"]
Oct 30 21:42:01.529221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.529196    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.547191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.547147    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"]
Oct 30 21:42:01.566494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.566459    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-64d142-serving-cert\""
Oct 30 21:42:01.638635 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.638599    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b988a3-a45c-4eb6-8324-7ea24b1aa434-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.638804 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.638651    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.638804 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.638730    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgxxm\" (UniqueName: \"kubernetes.io/projected/74b988a3-a45c-4eb6-8324-7ea24b1aa434-kube-api-access-lgxxm\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.638914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.638900    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.739896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.739856    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b988a3-a45c-4eb6-8324-7ea24b1aa434-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.740096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.739926    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.740096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.740039    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxxm\" (UniqueName: \"kubernetes.io/projected/74b988a3-a45c-4eb6-8324-7ea24b1aa434-kube-api-access-lgxxm\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.740251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.740239    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.740677 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.740658    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b988a3-a45c-4eb6-8324-7ea24b1aa434-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.742427 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.742407    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.742636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.742614    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.762665 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.762627    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxxm\" (UniqueName: \"kubernetes.io/projected/74b988a3-a45c-4eb6-8324-7ea24b1aa434-kube-api-access-lgxxm\") pod \"maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:01.844448 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:01.844346    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:02.182769 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:02.182727    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"]
Oct 30 21:42:02.185855 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:42:02.185826    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b988a3_a45c_4eb6_8324_7ea24b1aa434.slice/crio-93baf598ab60a8fa5ecb9080aba9484f63e5d921e0b1c564afe6bffb779f9e89 WatchSource:0}: Error finding container 93baf598ab60a8fa5ecb9080aba9484f63e5d921e0b1c564afe6bffb779f9e89: Status 404 returned error can't find the container with id 93baf598ab60a8fa5ecb9080aba9484f63e5d921e0b1c564afe6bffb779f9e89
Oct 30 21:42:03.196288 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:03.196254    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk" event={"ID":"74b988a3-a45c-4eb6-8324-7ea24b1aa434","Type":"ContainerStarted","Data":"5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d"}
Oct 30 21:42:03.196764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:03.196314    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk" event={"ID":"74b988a3-a45c-4eb6-8324-7ea24b1aa434","Type":"ContainerStarted","Data":"93baf598ab60a8fa5ecb9080aba9484f63e5d921e0b1c564afe6bffb779f9e89"}
Oct 30 21:42:03.196764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:03.196363    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:03.239298 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:03.239244    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk" podStartSLOduration=1.729586475 podStartE2EDuration="2.239229229s" totalImagesPullingTime="509.642754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:03.235012062 +0000 UTC m=+1923.364263521" watchObservedRunningTime="2026-10-30 21:42:03.239229229 +0000 UTC m=+1923.368480685"
Oct 30 21:42:09.211051 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:09.211018    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 21:42:11.030907 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:42:11.030871    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:42:12.804720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:12.804678    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"]
Oct 30 21:42:12.805366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:12.804933    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy" containerID="cri-o://30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92" gracePeriod=30
Oct 30 21:42:18.144514 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.144486    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:42:18.313417 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.313376    5886 generic.go:356] "Generic (PLEG): container finished" podID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerID="30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92" exitCode=0
Oct 30 21:42:18.313620 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.313443    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"
Oct 30 21:42:18.313620 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.313454    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" event={"ID":"82347ec2-49bc-45b7-83d7-4c57b829e416","Type":"ContainerDied","Data":"30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92"}
Oct 30 21:42:18.313620 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.313545    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb" event={"ID":"82347ec2-49bc-45b7-83d7-4c57b829e416","Type":"ContainerDied","Data":"4ccd17934fd3c809382f016158c8a340a7d80a54b21a88211b223891054e1f30"}
Oct 30 21:42:18.313620 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.313569    5886 scope.go:122] "RemoveContainer" containerID="30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92"
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332590    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-certs\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332640    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-podinfo\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332745    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-kube-api-access-zvmrn\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-kube-api-access-zvmrn\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332803    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-socket\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332835    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-token\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-token\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332868    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-envoy\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332904    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-credential-socket\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332956    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82347ec2-49bc-45b7-83d7-4c57b829e416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82347ec2-49bc-45b7-83d7-4c57b829e416-istiod-ca-cert\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.332997    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-data\") pod \"82347ec2-49bc-45b7-83d7-4c57b829e416\" (UID: \"82347ec2-49bc-45b7-83d7-4c57b829e416\") "
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.334238    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-socket" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:18.335515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.334486    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-certs" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:18.341130 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.341094    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-data" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:18.341567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.341536    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-credential-socket" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:18.341849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.341827    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82347ec2-49bc-45b7-83d7-4c57b829e416-istiod-ca-cert" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:18.350675 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.350641    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-envoy" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:18.352742 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.352713    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-podinfo" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:42:18.353065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.353033    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-kube-api-access-zvmrn" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "kube-api-access-zvmrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:18.356683 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.356657    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-token" pod "82347ec2-49bc-45b7-83d7-4c57b829e416" (UID: "82347ec2-49bc-45b7-83d7-4c57b829e416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:18.395279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.395064    5886 scope.go:122] "RemoveContainer" containerID="30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92"
Oct 30 21:42:18.402669 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:42:18.402576    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92\": container with ID starting with 30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92 not found: ID does not exist" containerID="30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92"
Oct 30 21:42:18.402669 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.402625    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92"} err="failed to get container status \"30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92\": rpc error: code = NotFound desc = could not find container \"30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92\": container with ID starting with 30d65d3cfe41bca0d5663a1659d331d29e240fe723fd7f569c8417e80ef84f92 not found: ID does not exist"
Oct 30 21:42:18.435295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435265    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvmrn\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-kube-api-access-zvmrn\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.435295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435292    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.435547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435308    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.435547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435321    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.435547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435331    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.435547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435339    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82347ec2-49bc-45b7-83d7-4c57b829e416-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.435547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435347    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.435547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435356    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82347ec2-49bc-45b7-83d7-4c57b829e416-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.435547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.435363    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82347ec2-49bc-45b7-83d7-4c57b829e416-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:18.658123 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.658094    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"]
Oct 30 21:42:18.691178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:18.688181    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-zbgkb"]
Oct 30 21:42:20.031846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:20.031816    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" path="/var/lib/kubelet/pods/82347ec2-49bc-45b7-83d7-4c57b829e416/volumes"
Oct 30 21:42:26.455432 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:26.455347    5886 generic.go:356] "Generic (PLEG): container finished" podID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerID="c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4" exitCode=0
Oct 30 21:42:26.456409 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:26.455869    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" event={"ID":"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7","Type":"ContainerDied","Data":"c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4"}
Oct 30 21:42:26.457466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:26.457448    5886 scope.go:122] "RemoveContainer" containerID="c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4"
Oct 30 21:42:26.477197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:26.460251    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:26.477197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:26.460349    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:26.477197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:26.460390    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:42:27.476572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:27.476535    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" event={"ID":"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7","Type":"ContainerStarted","Data":"6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324"}
Oct 30 21:42:28.042873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:28.042838    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:42:28.043053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:28.043012    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:42:28.043111 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:28.043089    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:42:29.043077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:29.043036    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:42:29.043535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:29.043095    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:42:30.043817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:30.043667    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:42:30.043817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:30.043730    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:42:31.043208 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:31.043176    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:42:31.043457 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:31.043429    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:42:32.042791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:32.042741    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:42:32.043220 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:32.042815    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:42:33.048773 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:33.048741    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:42:33.049481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:33.049460    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:42:33.050219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:33.050202    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 21:42:33.863944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:33.863912    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"]
Oct 30 21:42:33.864137 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:33.864114    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct" podUID="cb3d7945-8373-4c0c-992f-8df400d9c6b0" containerName="maas-api" containerID="cri-o://b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d" gracePeriod=30
Oct 30 21:42:34.496418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.496390    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:42:34.555122 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.555057    5886 generic.go:356] "Generic (PLEG): container finished" podID="cb3d7945-8373-4c0c-992f-8df400d9c6b0" containerID="b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d" exitCode=0
Oct 30 21:42:34.555326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.555204    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"
Oct 30 21:42:34.555396 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.555340    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct" event={"ID":"cb3d7945-8373-4c0c-992f-8df400d9c6b0","Type":"ContainerDied","Data":"b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d"}
Oct 30 21:42:34.555467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.555401    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct" event={"ID":"cb3d7945-8373-4c0c-992f-8df400d9c6b0","Type":"ContainerDied","Data":"c6d0ccbba6c4a68223f92bc75b0c53187db01682c67351a6a7021c738e4190e9"}
Oct 30 21:42:34.555467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.555436    5886 scope.go:122] "RemoveContainer" containerID="b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d"
Oct 30 21:42:34.567451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.567425    5886 scope.go:122] "RemoveContainer" containerID="b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d"
Oct 30 21:42:34.567793 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:42:34.567767    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d\": container with ID starting with b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d not found: ID does not exist" containerID="b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d"
Oct 30 21:42:34.567915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.567812    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d"} err="failed to get container status \"b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d\": rpc error: code = NotFound desc = could not find container \"b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d\": container with ID starting with b8ad3d8079c66f79501fd8984f251d01ca391465790e317b5b5c2c57f5baea4d not found: ID does not exist"
Oct 30 21:42:34.681032 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.680998    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3d7945-8373-4c0c-992f-8df400d9c6b0-kube-api-access-bhlqg\" (UniqueName: \"kubernetes.io/projected/cb3d7945-8373-4c0c-992f-8df400d9c6b0-kube-api-access-bhlqg\") pod \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") "
Oct 30 21:42:34.681246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.681198    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-metrics-certs\") pod \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") "
Oct 30 21:42:34.681309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.681260    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls\") pod \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") "
Oct 30 21:42:34.681434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.681416    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3d7945-8373-4c0c-992f-8df400d9c6b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3d7945-8373-4c0c-992f-8df400d9c6b0-openshift-service-ca\") pod \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\" (UID: \"cb3d7945-8373-4c0c-992f-8df400d9c6b0\") "
Oct 30 21:42:34.681864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.681833    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3d7945-8373-4c0c-992f-8df400d9c6b0-openshift-service-ca" pod "cb3d7945-8373-4c0c-992f-8df400d9c6b0" (UID: "cb3d7945-8373-4c0c-992f-8df400d9c6b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:34.683579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.683550    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3d7945-8373-4c0c-992f-8df400d9c6b0-kube-api-access-bhlqg" pod "cb3d7945-8373-4c0c-992f-8df400d9c6b0" (UID: "cb3d7945-8373-4c0c-992f-8df400d9c6b0"). InnerVolumeSpecName "kube-api-access-bhlqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:34.683786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.683770    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls" pod "cb3d7945-8373-4c0c-992f-8df400d9c6b0" (UID: "cb3d7945-8373-4c0c-992f-8df400d9c6b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:34.686044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.686023    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-metrics-certs" pod "cb3d7945-8373-4c0c-992f-8df400d9c6b0" (UID: "cb3d7945-8373-4c0c-992f-8df400d9c6b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:34.782844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.782751    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3d7945-8373-4c0c-992f-8df400d9c6b0-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:34.782844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.782798    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhlqg\" (UniqueName: \"kubernetes.io/projected/cb3d7945-8373-4c0c-992f-8df400d9c6b0-kube-api-access-bhlqg\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:34.782844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.782811    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:34.782844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.782820    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3d7945-8373-4c0c-992f-8df400d9c6b0-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:42:34.908822 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.908784    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"]
Oct 30 21:42:34.919738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:34.919706    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c19b75-56b984cdcf-9kjct"]
Oct 30 21:42:35.988938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:42:35.988904    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3d7945-8373-4c0c-992f-8df400d9c6b0" path="/var/lib/kubelet/pods/cb3d7945-8373-4c0c-992f-8df400d9c6b0/volumes"
Oct 30 21:42:41.029884 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:42:41.029847    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:43:02.332790 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:02.332758    5886 scope.go:122] "RemoveContainer" containerID="39d99e22afb1bea2d08e88051457a96944fc661fdf12562e17b2d2df28c7c4aa"
Oct 30 21:43:11.046370 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:43:11.046176    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:43:28.948631 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.948592    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"]
Oct 30 21:43:28.949410 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.949392    5886 cpu_manager.go:414] "containerMap: removing container" podUID="cb3d7945-8373-4c0c-992f-8df400d9c6b0" containerName="maas-api"
Oct 30 21:43:28.949477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.949412    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3d7945-8373-4c0c-992f-8df400d9c6b0" containerName="maas-api"
Oct 30 21:43:28.949477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.949442    5886 cpu_manager.go:414] "containerMap: removing container" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy"
Oct 30 21:43:28.949477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.949448    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy"
Oct 30 21:43:28.949584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.949548    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3d7945-8373-4c0c-992f-8df400d9c6b0" containerName="maas-api"
Oct 30 21:43:28.949584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.949562    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="82347ec2-49bc-45b7-83d7-4c57b829e416" containerName="istio-proxy"
Oct 30 21:43:28.953279 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.953257    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:28.976094 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.976064    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"]
Oct 30 21:43:28.978443 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:28.978418    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-5494e8-openshift-default-dockercfg-jppjn\""
Oct 30 21:43:29.057414 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.057381    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-data\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.057588 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.057501    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-podinfo\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.057588 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.057578    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-credential-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.057752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.057729    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-certs\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.057835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.057818    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-token\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.057889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.057849    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9xw\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-kube-api-access-qt9xw\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.058023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.058002    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.058098 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.058084    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b1ff52-2fef-4268-93b4-52325d4db4ff-istiod-ca-cert\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.058135 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.058123    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-envoy\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159382 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159347    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-podinfo\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159417    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-credential-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159456    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-certs\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159495    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-token\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159605    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9xw\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-kube-api-access-qt9xw\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159752 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159698    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159774    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b1ff52-2fef-4268-93b4-52325d4db4ff-istiod-ca-cert\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159816    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-envoy\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.159866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159858    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-data\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.160033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159905    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-certs\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.160033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.159966    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-credential-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.160194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.160144    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-socket\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.160329 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.160304    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-data\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.160610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.160590    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b1ff52-2fef-4268-93b4-52325d4db4ff-istiod-ca-cert\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.162002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.161978    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-envoy\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.162224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.162206    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-podinfo\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.181486 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.181456    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9xw\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-kube-api-access-qt9xw\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.181767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.181748    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-token\") pod \"e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") " pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.268293 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.268204    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:29.596726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.596684    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"]
Oct 30 21:43:29.599715 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:43:29.599675    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b1ff52_2fef_4268_93b4_52325d4db4ff.slice/crio-42da3d2a4f866814c54fbe5c08811a72aa068b44df57b16ab4fda6697af65de7 WatchSource:0}: Error finding container 42da3d2a4f866814c54fbe5c08811a72aa068b44df57b16ab4fda6697af65de7: Status 404 returned error can't find the container with id 42da3d2a4f866814c54fbe5c08811a72aa068b44df57b16ab4fda6697af65de7
Oct 30 21:43:29.602835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.602803    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:43:29.602916 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.602867    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:43:29.602916 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:29.602897    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:43:30.100868 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:30.100825    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" event={"ID":"57b1ff52-2fef-4268-93b4-52325d4db4ff","Type":"ContainerStarted","Data":"d53d45a09b4e8b72c0b60cd84ed451516a148a579965a20239b806cbc18ffffa"}
Oct 30 21:43:30.101383 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:30.100893    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" event={"ID":"57b1ff52-2fef-4268-93b4-52325d4db4ff","Type":"ContainerStarted","Data":"42da3d2a4f866814c54fbe5c08811a72aa068b44df57b16ab4fda6697af65de7"}
Oct 30 21:43:30.144442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:30.144382    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podStartSLOduration=2.144362005 podStartE2EDuration="2.144362005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:30.142111609 +0000 UTC m=+2010.271363067" watchObservedRunningTime="2026-10-30 21:43:30.144362005 +0000 UTC m=+2010.273613463"
Oct 30 21:43:30.272079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:30.272045    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:30.272450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:30.272426    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:43:30.272623 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:30.272599    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 21:43:31.273676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:31.273639    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:43:31.274314 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:31.274255    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 21:43:32.281179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:32.270534    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:43:32.281179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:32.270594    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 21:43:33.270568 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:33.270525    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:43:33.270930 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:33.270899    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 21:43:34.271177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:34.269506    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:43:34.271177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:34.269579    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 21:43:35.271540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:35.271502    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 21:43:35.272131 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:35.272098    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 21:43:36.274504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:36.274467    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:36.275017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:36.274842    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:36.275650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:36.275628    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 21:43:36.957165 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:36.957115    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"]
Oct 30 21:43:36.957462 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:36.957432    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29" podUID="9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" containerName="maas-api" containerID="cri-o://bf88b63ff3e30e9638d720b50eaf8a8447116c6a61c38f882b6aabdb4af9c858" gracePeriod=30
Oct 30 21:43:37.230220 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.229277    5886 generic.go:356] "Generic (PLEG): container finished" podID="9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" containerID="bf88b63ff3e30e9638d720b50eaf8a8447116c6a61c38f882b6aabdb4af9c858" exitCode=0
Oct 30 21:43:37.230220 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.229392    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29" event={"ID":"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55","Type":"ContainerDied","Data":"bf88b63ff3e30e9638d720b50eaf8a8447116c6a61c38f882b6aabdb4af9c858"}
Oct 30 21:43:37.308065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.308038    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:43:37.454208 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.454178    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-maas-api-tls\") pod \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") "
Oct 30 21:43:37.454391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.454230    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-metrics-certs\") pod \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") "
Oct 30 21:43:37.454391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.454362    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-openshift-service-ca\") pod \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") "
Oct 30 21:43:37.454466 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.454402    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-kube-api-access-pw742\" (UniqueName: \"kubernetes.io/projected/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-kube-api-access-pw742\") pod \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\" (UID: \"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55\") "
Oct 30 21:43:37.454707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.454683    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-openshift-service-ca" pod "9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" (UID: "9a355373-35ab-4f05-b1c4-4c3eaa5c8b55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:37.455060 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.455036    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:43:37.456618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.456587    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-kube-api-access-pw742" pod "9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" (UID: "9a355373-35ab-4f05-b1c4-4c3eaa5c8b55"). InnerVolumeSpecName "kube-api-access-pw742". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:37.456618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.456589    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-maas-api-tls" pod "9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" (UID: "9a355373-35ab-4f05-b1c4-4c3eaa5c8b55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:37.456618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.456578    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-metrics-certs" pod "9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" (UID: "9a355373-35ab-4f05-b1c4-4c3eaa5c8b55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:37.558391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.557888    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw742\" (UniqueName: \"kubernetes.io/projected/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-kube-api-access-pw742\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:43:37.558391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.557934    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:43:37.558391 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:37.557949    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:43:38.237515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:38.237481    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"
Oct 30 21:43:38.237698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:38.237514    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29" event={"ID":"9a355373-35ab-4f05-b1c4-4c3eaa5c8b55","Type":"ContainerDied","Data":"769722d45fc6419c60bc29115da0ac30c66c0f6ed7f822546861d3791adfa9ff"}
Oct 30 21:43:38.237698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:38.237574    5886 scope.go:122] "RemoveContainer" containerID="bf88b63ff3e30e9638d720b50eaf8a8447116c6a61c38f882b6aabdb4af9c858"
Oct 30 21:43:38.289722 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:38.289646    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"]
Oct 30 21:43:38.300341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:38.300317    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-7a4de0-cbb464778-qrc29"]
Oct 30 21:43:39.989856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:39.989816    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" path="/var/lib/kubelet/pods/9a355373-35ab-4f05-b1c4-4c3eaa5c8b55/volumes"
Oct 30 21:43:41.029594 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:43:41.029555    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:43:57.439019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.438979    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"]
Oct 30 21:43:57.439990 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.439968    5886 cpu_manager.go:414] "containerMap: removing container" podUID="9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" containerName="maas-api"
Oct 30 21:43:57.440065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.439992    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" containerName="maas-api"
Oct 30 21:43:57.440240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.440227    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a355373-35ab-4f05-b1c4-4c3eaa5c8b55" containerName="maas-api"
Oct 30 21:43:57.444361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.444334    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.463000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.462970    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a2fc47-openshift-default-dockercfg-59wcg\""
Oct 30 21:43:57.492854 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.492820    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8301798-2880-481f-ad6d-189c09446234-istiod-ca-cert\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.493023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.492936    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-istio-token\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.493023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.492986    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9lj8\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-kube-api-access-m9lj8\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.493132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.493114    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-socket\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.493225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.493209    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-data\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.493309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.493297    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8301798-2880-481f-ad6d-189c09446234-istio-podinfo\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.493366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.493352    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-envoy\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.493445 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.493413    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-certs\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.493521 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.493491    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-credential-socket\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.558783 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.558739    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"]
Oct 30 21:43:57.619180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.598093    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-data\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.669387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.632475    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-data\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.670702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.670274    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8301798-2880-481f-ad6d-189c09446234-istio-podinfo\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.671094 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.671074    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-envoy\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.671783 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.671762    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-certs\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.672412 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.672392    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-credential-socket\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.673000 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.672307    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-certs\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.673177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.672867    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-credential-socket\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.690182 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.688079    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8301798-2880-481f-ad6d-189c09446234-istio-podinfo\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.690418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.690058    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8301798-2880-481f-ad6d-189c09446234-istiod-ca-cert\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.690786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.690768    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-envoy\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.690973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.690949    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8301798-2880-481f-ad6d-189c09446234-istiod-ca-cert\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.691202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.691182    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-istio-token\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.691735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.691716    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9lj8\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-kube-api-access-m9lj8\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.692294 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.692276    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-socket\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.694552 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.694516    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-socket\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.721973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.721947    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-istio-token\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.721973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.721970    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9lj8\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-kube-api-access-m9lj8\") pod \"e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") " pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:57.761900 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:57.761875    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:58.110943 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:58.110915    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"]
Oct 30 21:43:58.111899 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:43:58.111876    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8301798_2880_481f_ad6d_189c09446234.slice/crio-a60f50340d16ae1477417be054b69ee4aad677806a30a7f7af5b3e2a3db9e6c2 WatchSource:0}: Error finding container a60f50340d16ae1477417be054b69ee4aad677806a30a7f7af5b3e2a3db9e6c2: Status 404 returned error can't find the container with id a60f50340d16ae1477417be054b69ee4aad677806a30a7f7af5b3e2a3db9e6c2
Oct 30 21:43:58.114762 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:58.114731    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:43:58.114904 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:58.114801    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:43:58.114904 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:58.114832    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:43:58.390221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:58.387444    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" event={"ID":"f8301798-2880-481f-ad6d-189c09446234","Type":"ContainerStarted","Data":"a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa"}
Oct 30 21:43:58.390221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:58.387503    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" event={"ID":"f8301798-2880-481f-ad6d-189c09446234","Type":"ContainerStarted","Data":"a60f50340d16ae1477417be054b69ee4aad677806a30a7f7af5b3e2a3db9e6c2"}
Oct 30 21:43:58.445371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:58.445312    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" podStartSLOduration=1.4451123209999999 podStartE2EDuration="1.445112321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:58.440950953 +0000 UTC m=+2038.570202409" watchObservedRunningTime="2026-10-30 21:43:58.445112321 +0000 UTC m=+2038.574363781"
Oct 30 21:43:58.762923 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:58.762890    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:43:59.769723 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:43:59.769686    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:44:00.410537 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:00.410499    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:44:00.411939 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:00.411907    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:44:11.029890 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:44:11.029848    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:44:24.639951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.639915    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"]
Oct 30 21:44:24.644530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.644505    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.661739 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.661702    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-a2fc47-serving-cert\""
Oct 30 21:44:24.696220 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.675884    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"]
Oct 30 21:44:24.787931 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.787883    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fjj\" (UniqueName: \"kubernetes.io/projected/00fde4c1-ee1c-4223-8c68-988c4058a07b-kube-api-access-h9fjj\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.788173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.787942    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-metrics-certs\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.788173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.788107    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fde4c1-ee1c-4223-8c68-988c4058a07b-openshift-service-ca\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.788291 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.788226    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-maas-api-tls\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.898229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.893251    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fjj\" (UniqueName: \"kubernetes.io/projected/00fde4c1-ee1c-4223-8c68-988c4058a07b-kube-api-access-h9fjj\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.898229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.893324    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-metrics-certs\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.898229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.893390    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fde4c1-ee1c-4223-8c68-988c4058a07b-openshift-service-ca\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.898229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.893454    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-maas-api-tls\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.915178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.905937    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-maas-api-tls\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.915178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.906685    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fde4c1-ee1c-4223-8c68-988c4058a07b-openshift-service-ca\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.936246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.932488    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-metrics-certs\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.942188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.939902    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fjj\" (UniqueName: \"kubernetes.io/projected/00fde4c1-ee1c-4223-8c68-988c4058a07b-kube-api-access-h9fjj\") pod \"maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:24.962183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:24.960345    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:25.299363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:25.299328    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"]
Oct 30 21:44:25.303453 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:44:25.303422    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fde4c1_ee1c_4223_8c68_988c4058a07b.slice/crio-6c02779875d2466aef57eed4dc37567f84146b518c6c22914db630f45bd96a9d WatchSource:0}: Error finding container 6c02779875d2466aef57eed4dc37567f84146b518c6c22914db630f45bd96a9d: Status 404 returned error can't find the container with id 6c02779875d2466aef57eed4dc37567f84146b518c6c22914db630f45bd96a9d
Oct 30 21:44:25.304547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:25.304521    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:44:25.641010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:25.640921    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49" event={"ID":"00fde4c1-ee1c-4223-8c68-988c4058a07b","Type":"ContainerStarted","Data":"6c02779875d2466aef57eed4dc37567f84146b518c6c22914db630f45bd96a9d"}
Oct 30 21:44:26.650542 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:26.650505    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49" event={"ID":"00fde4c1-ee1c-4223-8c68-988c4058a07b","Type":"ContainerStarted","Data":"6785c118f8259a07447924c4a0ecb59f9ab70e150f1c98263cff501f7cb3c20c"}
Oct 30 21:44:26.651652 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:26.651627    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:26.690907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:26.690839    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49" podStartSLOduration=2.307958228 podStartE2EDuration="2.690816618s" totalImagesPullingTime="382.85839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:26.688107278 +0000 UTC m=+2066.817358758" watchObservedRunningTime="2026-10-30 21:44:26.690816618 +0000 UTC m=+2066.820068078"
Oct 30 21:44:33.711024 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:33.710993    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:44:41.029141 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:44:41.029056    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:44:56.995510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:56.995475    5886 generic.go:356] "Generic (PLEG): container finished" podID="f8301798-2880-481f-ad6d-189c09446234" containerID="a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa" exitCode=0
Oct 30 21:44:56.996077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:56.995517    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" event={"ID":"f8301798-2880-481f-ad6d-189c09446234","Type":"ContainerDied","Data":"a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa"}
Oct 30 21:44:56.996077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:56.996006    5886 scope.go:122] "RemoveContainer" containerID="a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa"
Oct 30 21:44:56.997679 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:56.997650    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:44:56.997806 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:56.997736    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:44:56.997806 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:56.997781    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:44:58.014642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:58.011950    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" event={"ID":"f8301798-2880-481f-ad6d-189c09446234","Type":"ContainerStarted","Data":"7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c"}
Oct 30 21:44:58.771887 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:58.771855    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:44:58.772272 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:58.772248    5886 patch_prober.go:29] interesting pod/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:44:58.772446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:58.772422    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:44:59.763545 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:59.763505    5886 patch_prober.go:29] interesting pod/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:44:59.764193 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:44:59.764140    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:45:00.335247 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.332214    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"]
Oct 30 21:45:00.344877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.343373    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"]
Oct 30 21:45:00.344877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.343489    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:00.412304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.412273    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:45:00.430977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.430950    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 21:45:00.481179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.476145    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw\" (UID: \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:00.481179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.476465    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rqr\" (UniqueName: \"kubernetes.io/projected/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-kube-api-access-f7rqr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw\" (UID: \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:00.594250 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.593559    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw\" (UID: \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:00.594250 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.593843    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rqr\" (UniqueName: \"kubernetes.io/projected/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-kube-api-access-f7rqr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw\" (UID: \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:00.595240 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.594988    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw\" (UID: \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:00.613664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.613633    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rqr\" (UniqueName: \"kubernetes.io/projected/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-kube-api-access-f7rqr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw\" (UID: \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:00.667360 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.666388    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:00.765318 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.762413    5886 patch_prober.go:29] interesting pod/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:45:00.765318 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:00.762473    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:45:01.041915 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:01.039131    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"]
Oct 30 21:45:01.042033 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:45:01.041998    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fb635f_90e4_49e9_9a8f_f4e185a8a50e.slice/crio-d0f104650d5c28e459a799412e2b5a5adc888ded8655bdae005c84af388da491 WatchSource:0}: Error finding container d0f104650d5c28e459a799412e2b5a5adc888ded8655bdae005c84af388da491: Status 404 returned error can't find the container with id d0f104650d5c28e459a799412e2b5a5adc888ded8655bdae005c84af388da491
Oct 30 21:45:01.792200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:01.791329    5886 patch_prober.go:29] interesting pod/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:45:01.792200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:01.791422    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:45:02.044995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:02.044904    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" event={"ID":"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e","Type":"ContainerStarted","Data":"d0f104650d5c28e459a799412e2b5a5adc888ded8655bdae005c84af388da491"}
Oct 30 21:45:02.792625 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:02.792584    5886 patch_prober.go:29] interesting pod/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:45:02.793084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:02.793049    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:45:03.768479 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:03.768435    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:45:03.770101 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:03.770072    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:45:05.074644 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:05.074596    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" event={"ID":"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e","Type":"ContainerStarted","Data":"0a8d3614546e04c981d67d05a68367b58e71c7b2079315469af3390ef0c4560a"}
Oct 30 21:45:05.129061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:05.128823    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" podStartSLOduration=1.861777762 podStartE2EDuration="5.12880028s" totalImagesPullingTime="3.267022518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:05.127294111 +0000 UTC m=+2105.256545567" watchObservedRunningTime="2026-10-30 21:45:05.12880028 +0000 UTC m=+2105.258051738"
Oct 30 21:45:06.083315 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:06.083276    5886 generic.go:356] "Generic (PLEG): container finished" podID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerID="0a8d3614546e04c981d67d05a68367b58e71c7b2079315469af3390ef0c4560a" exitCode=60
Oct 30 21:45:06.083749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:06.083359    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" event={"ID":"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e","Type":"ContainerDied","Data":"0a8d3614546e04c981d67d05a68367b58e71c7b2079315469af3390ef0c4560a"}
Oct 30 21:45:06.083749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:06.083640    5886 scope.go:122] "RemoveContainer" containerID="0a8d3614546e04c981d67d05a68367b58e71c7b2079315469af3390ef0c4560a"
Oct 30 21:45:07.140549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:07.140502    5886 generic.go:356] "Generic (PLEG): container finished" podID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerID="e99a71a8889c6a87cfe6af924e3eda6804990250ddb1d86454eb83c6cd4607db" exitCode=60
Oct 30 21:45:07.141097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:07.140707    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" event={"ID":"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e","Type":"ContainerDied","Data":"e99a71a8889c6a87cfe6af924e3eda6804990250ddb1d86454eb83c6cd4607db"}
Oct 30 21:45:07.141097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:07.140774    5886 scope.go:122] "RemoveContainer" containerID="0a8d3614546e04c981d67d05a68367b58e71c7b2079315469af3390ef0c4560a"
Oct 30 21:45:07.151541 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:07.151507    5886 scope.go:122] "RemoveContainer" containerID="e99a71a8889c6a87cfe6af924e3eda6804990250ddb1d86454eb83c6cd4607db"
Oct 30 21:45:07.151983 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:45:07.151954    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw_odh-ai-gateway-infra(a9fb635f-90e4-49e9-9a8f-f4e185a8a50e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e"
Oct 30 21:45:08.149744 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:08.149712    5886 scope.go:122] "RemoveContainer" containerID="e99a71a8889c6a87cfe6af924e3eda6804990250ddb1d86454eb83c6cd4607db"
Oct 30 21:45:08.150222 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:45:08.150125    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw_odh-ai-gateway-infra(a9fb635f-90e4-49e9-9a8f-f4e185a8a50e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e"
Oct 30 21:45:11.030675 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:45:11.030635    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:45:19.986198 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:19.986168    5886 scope.go:122] "RemoveContainer" containerID="e99a71a8889c6a87cfe6af924e3eda6804990250ddb1d86454eb83c6cd4607db"
Oct 30 21:45:20.288832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:20.288794    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" event={"ID":"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e","Type":"ContainerStarted","Data":"a1149a0fc01f7e0f4887c0478b788dbb07f8cc90271333f4efd0a60864536056"}
Oct 30 21:45:21.300047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:21.300009    5886 generic.go:356] "Generic (PLEG): container finished" podID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerID="a1149a0fc01f7e0f4887c0478b788dbb07f8cc90271333f4efd0a60864536056" exitCode=60
Oct 30 21:45:21.300554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:21.300097    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" event={"ID":"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e","Type":"ContainerDied","Data":"a1149a0fc01f7e0f4887c0478b788dbb07f8cc90271333f4efd0a60864536056"}
Oct 30 21:45:21.300554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:21.300192    5886 scope.go:122] "RemoveContainer" containerID="e99a71a8889c6a87cfe6af924e3eda6804990250ddb1d86454eb83c6cd4607db"
Oct 30 21:45:21.300554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:21.300543    5886 scope.go:122] "RemoveContainer" containerID="a1149a0fc01f7e0f4887c0478b788dbb07f8cc90271333f4efd0a60864536056"
Oct 30 21:45:21.300934 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:45:21.300909    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw_odh-ai-gateway-infra(a9fb635f-90e4-49e9-9a8f-f4e185a8a50e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e"
Oct 30 21:45:21.337758 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:21.337726    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"]
Oct 30 21:45:22.539423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:22.539395    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:22.707943 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:22.707908    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-kube-api-access-f7rqr\" (UniqueName: \"kubernetes.io/projected/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-kube-api-access-f7rqr\") pod \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\" (UID: \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\") "
Oct 30 21:45:22.708217 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:22.707998    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-maas-api-service-ca\") pod \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\" (UID: \"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e\") "
Oct 30 21:45:22.708383 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:22.708355    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-maas-api-service-ca" pod "a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" (UID: "a9fb635f-90e4-49e9-9a8f-f4e185a8a50e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:22.708700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:22.708684    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:22.710132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:22.710109    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-kube-api-access-f7rqr" pod "a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" (UID: "a9fb635f-90e4-49e9-9a8f-f4e185a8a50e"). InnerVolumeSpecName "kube-api-access-f7rqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:22.810566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:22.810533    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7rqr\" (UniqueName: \"kubernetes.io/projected/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e-kube-api-access-f7rqr\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:23.319133 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:23.319104    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"
Oct 30 21:45:23.319341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:23.319133    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw" event={"ID":"a9fb635f-90e4-49e9-9a8f-f4e185a8a50e","Type":"ContainerDied","Data":"d0f104650d5c28e459a799412e2b5a5adc888ded8655bdae005c84af388da491"}
Oct 30 21:45:23.319341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:23.319204    5886 scope.go:122] "RemoveContainer" containerID="a1149a0fc01f7e0f4887c0478b788dbb07f8cc90271333f4efd0a60864536056"
Oct 30 21:45:23.371392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:23.371365    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"]
Oct 30 21:45:23.382271 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:23.382245    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889945-78gmw"]
Oct 30 21:45:24.028931 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:24.028887    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" path="/var/lib/kubelet/pods/a9fb635f-90e4-49e9-9a8f-f4e185a8a50e/volumes"
Oct 30 21:45:29.019407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:29.019346    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"]
Oct 30 21:45:29.019962 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:29.019675    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" containerID="cri-o://cb991ccbc1242c6e6948e2e4f96fa17f56a67cbcd7a1ac2ad088b6627da65e92" gracePeriod=30
Oct 30 21:45:33.207842 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:33.207799    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-2bqcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:45:33.208345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:33.207879    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:45:34.419003 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.418961    5886 generic.go:356] "Generic (PLEG): container finished" podID="9c342234-2531-49aa-b054-e7be98b2b36d" containerID="cb991ccbc1242c6e6948e2e4f96fa17f56a67cbcd7a1ac2ad088b6627da65e92" exitCode=0
Oct 30 21:45:34.419484 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.419138    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" event={"ID":"9c342234-2531-49aa-b054-e7be98b2b36d","Type":"ContainerDied","Data":"cb991ccbc1242c6e6948e2e4f96fa17f56a67cbcd7a1ac2ad088b6627da65e92"}
Oct 30 21:45:34.465625 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.465598    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:45:34.603277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603190    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c342234-2531-49aa-b054-e7be98b2b36d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c342234-2531-49aa-b054-e7be98b2b36d-istio-podinfo\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.603277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603249    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-data\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.603483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603414    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-credential-socket\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.603548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603527    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-certs\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.603607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603524    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-data" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:34.603664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603584    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-istio-token\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-istio-token\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.603722 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603690    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-socket\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.603781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603698    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-credential-socket" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:34.603781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603729    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-envoy\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.603781 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603733    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-certs" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:34.603936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603806    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-kube-api-access-822k4\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-kube-api-access-822k4\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.603936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603918    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-socket" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:34.604029 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.603938    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c342234-2531-49aa-b054-e7be98b2b36d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c342234-2531-49aa-b054-e7be98b2b36d-istiod-ca-cert\") pod \"9c342234-2531-49aa-b054-e7be98b2b36d\" (UID: \"9c342234-2531-49aa-b054-e7be98b2b36d\") "
Oct 30 21:45:34.604405 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.604362    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c342234-2531-49aa-b054-e7be98b2b36d-istiod-ca-cert" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:34.604670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.604651    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:34.604670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.604671    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c342234-2531-49aa-b054-e7be98b2b36d-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:34.604670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.604681    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:34.604926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.604695    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:34.604926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.604708    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:34.605817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.605793    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c342234-2531-49aa-b054-e7be98b2b36d-istio-podinfo" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:45:34.606022 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.606003    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-kube-api-access-822k4" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "kube-api-access-822k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:34.606124 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.606017    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-istio-token" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:34.606447 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.606424    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-envoy" pod "9c342234-2531-49aa-b054-e7be98b2b36d" (UID: "9c342234-2531-49aa-b054-e7be98b2b36d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:34.727202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.714412    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:34.727202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.714453    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c342234-2531-49aa-b054-e7be98b2b36d-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:34.727202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.714472    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-822k4\" (UniqueName: \"kubernetes.io/projected/9c342234-2531-49aa-b054-e7be98b2b36d-kube-api-access-822k4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:34.727202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:34.714488    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c342234-2531-49aa-b054-e7be98b2b36d-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:35.427817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:35.427783    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"
Oct 30 21:45:35.428393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:35.427813    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx" event={"ID":"9c342234-2531-49aa-b054-e7be98b2b36d","Type":"ContainerDied","Data":"c6aaa3312d3b50406ba792480e91d1f7e08502c0ee4791c1184a5def6be709ea"}
Oct 30 21:45:35.428393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:35.427888    5886 scope.go:122] "RemoveContainer" containerID="cb991ccbc1242c6e6948e2e4f96fa17f56a67cbcd7a1ac2ad088b6627da65e92"
Oct 30 21:45:35.489119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:35.489085    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"]
Oct 30 21:45:35.500373 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:35.500342    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-2bqcx"]
Oct 30 21:45:35.991327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:35.991288    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" path="/var/lib/kubelet/pods/9c342234-2531-49aa-b054-e7be98b2b36d/volumes"
Oct 30 21:45:40.230711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.230672    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"]
Oct 30 21:45:40.231107 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.230866    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49" podUID="00fde4c1-ee1c-4223-8c68-988c4058a07b" containerName="maas-api" containerID="cri-o://6785c118f8259a07447924c4a0ecb59f9ab70e150f1c98263cff501f7cb3c20c" gracePeriod=30
Oct 30 21:45:40.584248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.582352    5886 generic.go:356] "Generic (PLEG): container finished" podID="00fde4c1-ee1c-4223-8c68-988c4058a07b" containerID="6785c118f8259a07447924c4a0ecb59f9ab70e150f1c98263cff501f7cb3c20c" exitCode=0
Oct 30 21:45:40.584248 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.582396    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49" event={"ID":"00fde4c1-ee1c-4223-8c68-988c4058a07b","Type":"ContainerDied","Data":"6785c118f8259a07447924c4a0ecb59f9ab70e150f1c98263cff501f7cb3c20c"}
Oct 30 21:45:40.642125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.641440    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:45:40.824320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.824233    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fde4c1-ee1c-4223-8c68-988c4058a07b-kube-api-access-h9fjj\" (UniqueName: \"kubernetes.io/projected/00fde4c1-ee1c-4223-8c68-988c4058a07b-kube-api-access-h9fjj\") pod \"00fde4c1-ee1c-4223-8c68-988c4058a07b\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") "
Oct 30 21:45:40.824510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.824346    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-metrics-certs\") pod \"00fde4c1-ee1c-4223-8c68-988c4058a07b\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") "
Oct 30 21:45:40.824510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.824462    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-maas-api-tls\") pod \"00fde4c1-ee1c-4223-8c68-988c4058a07b\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") "
Oct 30 21:45:40.824619 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.824508    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fde4c1-ee1c-4223-8c68-988c4058a07b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fde4c1-ee1c-4223-8c68-988c4058a07b-openshift-service-ca\") pod \"00fde4c1-ee1c-4223-8c68-988c4058a07b\" (UID: \"00fde4c1-ee1c-4223-8c68-988c4058a07b\") "
Oct 30 21:45:40.828384 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.826138    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fde4c1-ee1c-4223-8c68-988c4058a07b-openshift-service-ca" pod "00fde4c1-ee1c-4223-8c68-988c4058a07b" (UID: "00fde4c1-ee1c-4223-8c68-988c4058a07b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:40.840476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.839981    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fde4c1-ee1c-4223-8c68-988c4058a07b-kube-api-access-h9fjj" pod "00fde4c1-ee1c-4223-8c68-988c4058a07b" (UID: "00fde4c1-ee1c-4223-8c68-988c4058a07b"). InnerVolumeSpecName "kube-api-access-h9fjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:40.840476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.840129    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-maas-api-tls" pod "00fde4c1-ee1c-4223-8c68-988c4058a07b" (UID: "00fde4c1-ee1c-4223-8c68-988c4058a07b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:40.842430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.842148    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-metrics-certs" pod "00fde4c1-ee1c-4223-8c68-988c4058a07b" (UID: "00fde4c1-ee1c-4223-8c68-988c4058a07b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:40.926173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.926125    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9fjj\" (UniqueName: \"kubernetes.io/projected/00fde4c1-ee1c-4223-8c68-988c4058a07b-kube-api-access-h9fjj\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:40.926173 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.926171    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:40.926385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.926189    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00fde4c1-ee1c-4223-8c68-988c4058a07b-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:40.926385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:40.926201    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00fde4c1-ee1c-4223-8c68-988c4058a07b-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:41.031200 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:45:41.031125    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:45:41.599036 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:41.599004    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"
Oct 30 21:45:41.599583 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:41.599036    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49" event={"ID":"00fde4c1-ee1c-4223-8c68-988c4058a07b","Type":"ContainerDied","Data":"6c02779875d2466aef57eed4dc37567f84146b518c6c22914db630f45bd96a9d"}
Oct 30 21:45:41.599583 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:41.599102    5886 scope.go:122] "RemoveContainer" containerID="6785c118f8259a07447924c4a0ecb59f9ab70e150f1c98263cff501f7cb3c20c"
Oct 30 21:45:41.652760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:41.652726    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"]
Oct 30 21:45:41.664123 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:41.664070    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-a2fc47-5db6449dc4-g2b49"]
Oct 30 21:45:41.990070 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:41.990032    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fde4c1-ee1c-4223-8c68-988c4058a07b" path="/var/lib/kubelet/pods/00fde4c1-ee1c-4223-8c68-988c4058a07b/volumes"
Oct 30 21:45:47.972819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:47.972779    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"]
Oct 30 21:45:47.974008 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:47.973937    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy" containerID="cri-o://7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c" gracePeriod=30
Oct 30 21:45:48.771914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:48.771849    5886 patch_prober.go:29] interesting pod/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:45:48.772120 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:48.771966    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:45:49.117088 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.116898    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"]
Oct 30 21:45:49.118103 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.117900    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:45:49.118301 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118286    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:45:49.118395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118376    5886 cpu_manager.go:414] "containerMap: removing container" podUID="00fde4c1-ee1c-4223-8c68-988c4058a07b" containerName="maas-api"
Oct 30 21:45:49.118395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118390    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fde4c1-ee1c-4223-8c68-988c4058a07b" containerName="maas-api"
Oct 30 21:45:49.118584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118453    5886 cpu_manager.go:414] "containerMap: removing container" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy"
Oct 30 21:45:49.118584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118480    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy"
Oct 30 21:45:49.118584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118510    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:45:49.118584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118515    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:45:49.118778 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118675    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fde4c1-ee1c-4223-8c68-988c4058a07b" containerName="maas-api"
Oct 30 21:45:49.118778 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118688    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:45:49.118778 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118706    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c342234-2531-49aa-b054-e7be98b2b36d" containerName="istio-proxy"
Oct 30 21:45:49.118778 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.118719    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:45:49.126710 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.126682    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.127952 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.127925    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"]
Oct 30 21:45:49.140104 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.140083    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-0d74b3-openshift-default-dockercfg-gp79q\""
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.285400    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm5m8\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-kube-api-access-wm5m8\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.285537    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-certs\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.285597    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-envoy\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.285631    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-data\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.285754    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istiod-ca-cert\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.285795    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-credential-socket\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.285827    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-socket\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.285936    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-podinfo\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.294175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.286011    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-token\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.386953    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-data\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387260 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387060    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istiod-ca-cert\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387260 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387101    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-credential-socket\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387260 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387129    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-socket\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387260 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387237    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-podinfo\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387524 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387325    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-token\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387524 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387392    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm5m8\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-kube-api-access-wm5m8\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387524 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387453    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-data\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387524 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387490    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-certs\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.387750 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.387540    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-envoy\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.388034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.388009    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istiod-ca-cert\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.388150 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.388127    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-credential-socket\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.388441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.388413    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-certs\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.388542 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.388487    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-socket\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.391124 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.391098    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-envoy\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.391761 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.391700    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-podinfo\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.422693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.422663    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-token\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.422858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.422824    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm5m8\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-kube-api-access-wm5m8\") pod \"e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") " pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.445330 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.445305    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:49.809876 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.809837    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"]
Oct 30 21:45:49.812664 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:45:49.812627    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5b82aa_2eac_4b99_9b14_088a0ff2a7c1.slice/crio-a043fbe27e00f0abb672223227e4c0a2cfc9ca692219a40e7e8e9037533ba346 WatchSource:0}: Error finding container a043fbe27e00f0abb672223227e4c0a2cfc9ca692219a40e7e8e9037533ba346: Status 404 returned error can't find the container with id a043fbe27e00f0abb672223227e4c0a2cfc9ca692219a40e7e8e9037533ba346
Oct 30 21:45:49.815623 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.815589    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:45:49.815713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.815652    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:45:49.815713 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:49.815681    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:45:50.670352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:50.670308    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp" event={"ID":"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1","Type":"ContainerStarted","Data":"2e25f9e9a4c36222fc6906178d63732ef6eb73d1852eebbfbfd23d840f9ea989"}
Oct 30 21:45:50.670914 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:50.670377    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp" event={"ID":"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1","Type":"ContainerStarted","Data":"a043fbe27e00f0abb672223227e4c0a2cfc9ca692219a40e7e8e9037533ba346"}
Oct 30 21:45:50.727553 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:50.727488    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp" podStartSLOduration=1.72746886 podStartE2EDuration="1.72746886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:50.723915762 +0000 UTC m=+2150.853167219" watchObservedRunningTime="2026-10-30 21:45:50.72746886 +0000 UTC m=+2150.856720318"
Oct 30 21:45:51.445709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:51.445675    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:51.462547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:51.462508    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:51.681120 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:51.681088    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:51.682046 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:51.682029    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:45:53.329407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.329383    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:45:53.474256 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474218    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-credential-socket\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.474434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474310    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8301798-2880-481f-ad6d-189c09446234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8301798-2880-481f-ad6d-189c09446234-istio-podinfo\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.474484 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474452    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-istio-token\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-istio-token\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.474517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474480    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-credential-socket" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:53.474562 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474547    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-socket\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.474607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474575    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-envoy\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.474658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474630    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-kube-api-access-m9lj8\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-kube-api-access-m9lj8\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.474719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474674    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-certs\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.474779 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474723    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-data\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.475057 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.474855    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8301798-2880-481f-ad6d-189c09446234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8301798-2880-481f-ad6d-189c09446234-istiod-ca-cert\") pod \"f8301798-2880-481f-ad6d-189c09446234\" (UID: \"f8301798-2880-481f-ad6d-189c09446234\") "
Oct 30 21:45:53.475218 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.475115    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-data" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:53.475372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.475339    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-certs" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:53.475578 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.475483    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8301798-2880-481f-ad6d-189c09446234-istiod-ca-cert" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:53.475726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.475634    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.475726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.475654    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.475726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.475668    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8301798-2880-481f-ad6d-189c09446234-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.475726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.475681    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.475985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.475958    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-socket" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:53.477361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.477333    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-kube-api-access-m9lj8" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "kube-api-access-m9lj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:53.477361 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.477347    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8301798-2880-481f-ad6d-189c09446234-istio-podinfo" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:45:53.477621 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.477600    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-envoy" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:53.477795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.477775    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-istio-token" pod "f8301798-2880-481f-ad6d-189c09446234" (UID: "f8301798-2880-481f-ad6d-189c09446234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:53.576787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.576701    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8301798-2880-481f-ad6d-189c09446234-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.576787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.576735    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.576787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.576744    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.576787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.576752    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8301798-2880-481f-ad6d-189c09446234-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.576787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.576761    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9lj8\" (UniqueName: \"kubernetes.io/projected/f8301798-2880-481f-ad6d-189c09446234-kube-api-access-m9lj8\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:45:53.732314 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.732273    5886 generic.go:356] "Generic (PLEG): container finished" podID="f8301798-2880-481f-ad6d-189c09446234" containerID="7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c" exitCode=0
Oct 30 21:45:53.732504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.732326    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"
Oct 30 21:45:53.732504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.732357    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" event={"ID":"f8301798-2880-481f-ad6d-189c09446234","Type":"ContainerDied","Data":"7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c"}
Oct 30 21:45:53.732504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.732413    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r" event={"ID":"f8301798-2880-481f-ad6d-189c09446234","Type":"ContainerDied","Data":"a60f50340d16ae1477417be054b69ee4aad677806a30a7f7af5b3e2a3db9e6c2"}
Oct 30 21:45:53.732504 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.732440    5886 scope.go:122] "RemoveContainer" containerID="7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c"
Oct 30 21:45:53.744138 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.744119    5886 scope.go:122] "RemoveContainer" containerID="a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa"
Oct 30 21:45:53.753400 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.753384    5886 scope.go:122] "RemoveContainer" containerID="7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c"
Oct 30 21:45:53.753670 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:45:53.753648    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c\": container with ID starting with 7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c not found: ID does not exist" containerID="7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c"
Oct 30 21:45:53.753722 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.753683    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c"} err="failed to get container status \"7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c\": rpc error: code = NotFound desc = could not find container \"7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c\": container with ID starting with 7c96a5d02289d2ba84a1fc662e9022b79ccf0009885d0c5434aea699c4198a3c not found: ID does not exist"
Oct 30 21:45:53.753722 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.753715    5886 scope.go:122] "RemoveContainer" containerID="a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa"
Oct 30 21:45:53.753945 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:45:53.753928    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa\": container with ID starting with a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa not found: ID does not exist" containerID="a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa"
Oct 30 21:45:53.753984 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.753950    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa"} err="failed to get container status \"a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa\": rpc error: code = NotFound desc = could not find container \"a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa\": container with ID starting with a1703716d1ded73f7594119c313c404148c2982f10626f1c2b40e2b4bd52b4aa not found: ID does not exist"
Oct 30 21:45:53.788481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.788423    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"]
Oct 30 21:45:53.795423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.795380    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a2fc47-openshift-default-5f9675966f-kbh8r"]
Oct 30 21:45:53.991963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:53.991918    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8301798-2880-481f-ad6d-189c09446234" path="/var/lib/kubelet/pods/f8301798-2880-481f-ad6d-189c09446234/volumes"
Oct 30 21:45:59.113200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:59.113150    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"]
Oct 30 21:45:59.113698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:45:59.113420    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy" containerID="cri-o://693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530" gracePeriod=30
Oct 30 21:46:04.450370 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.450342    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:46:04.561894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.561862    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-data\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562095 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.561917    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-kube-api-access-rft9l\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-kube-api-access-rft9l\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562095 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.561936    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-credential-socket\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562095 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562053    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-istio-token\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-istio-token\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562104    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-envoy\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562204    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-certs\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562366 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562304    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15707226-8100-4108-98f0-8697986f92af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15707226-8100-4108-98f0-8697986f92af-istiod-ca-cert\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562377    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15707226-8100-4108-98f0-8697986f92af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15707226-8100-4108-98f0-8697986f92af-istio-podinfo\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562202    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-data" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:04.562516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562224    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-credential-socket" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:04.562516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562504    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-socket\") pod \"15707226-8100-4108-98f0-8697986f92af\" (UID: \"15707226-8100-4108-98f0-8697986f92af\") "
Oct 30 21:46:04.562739 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.562720    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-certs" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:04.563379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.563064    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-socket" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:04.563379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.563144    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15707226-8100-4108-98f0-8697986f92af-istiod-ca-cert" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:04.563379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.563297    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.563379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.563314    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.563379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.563328    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.563379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.563341    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15707226-8100-4108-98f0-8697986f92af-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.563379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.563354    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.564764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.564734    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-envoy" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:04.564894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.564777    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15707226-8100-4108-98f0-8697986f92af-istio-podinfo" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:04.564894 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.564828    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-istio-token" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:04.565010 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.564992    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-kube-api-access-rft9l" pod "15707226-8100-4108-98f0-8697986f92af" (UID: "15707226-8100-4108-98f0-8697986f92af"). InnerVolumeSpecName "kube-api-access-rft9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:04.664770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.664729    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rft9l\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-kube-api-access-rft9l\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.664770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.664759    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15707226-8100-4108-98f0-8697986f92af-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.664770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.664770    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15707226-8100-4108-98f0-8697986f92af-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.664770 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.664777    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15707226-8100-4108-98f0-8697986f92af-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:04.832727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.832641    5886 generic.go:356] "Generic (PLEG): container finished" podID="15707226-8100-4108-98f0-8697986f92af" containerID="693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530" exitCode=0
Oct 30 21:46:04.832727 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.832716    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"
Oct 30 21:46:04.832934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.832726    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" event={"ID":"15707226-8100-4108-98f0-8697986f92af","Type":"ContainerDied","Data":"693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530"}
Oct 30 21:46:04.832934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.832782    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8" event={"ID":"15707226-8100-4108-98f0-8697986f92af","Type":"ContainerDied","Data":"10f8cb5b21caf1dae84658c1b5085253264e7b3833687f05e2ca1186dff24344"}
Oct 30 21:46:04.832934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.832809    5886 scope.go:122] "RemoveContainer" containerID="693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530"
Oct 30 21:46:04.844828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.844810    5886 scope.go:122] "RemoveContainer" containerID="693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530"
Oct 30 21:46:04.845142 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:46:04.845115    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530\": container with ID starting with 693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530 not found: ID does not exist" containerID="693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530"
Oct 30 21:46:04.845278 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.845151    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530"} err="failed to get container status \"693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530\": rpc error: code = NotFound desc = could not find container \"693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530\": container with ID starting with 693f168d061939af40f509b1d1dacfc398f1f28730a84263fc874fc880017530 not found: ID does not exist"
Oct 30 21:46:04.887107 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.887071    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"]
Oct 30 21:46:04.891817 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:04.891787    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-gf4z8"]
Oct 30 21:46:05.991346 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:05.991306    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15707226-8100-4108-98f0-8697986f92af" path="/var/lib/kubelet/pods/15707226-8100-4108-98f0-8697986f92af/volumes"
Oct 30 21:46:07.820437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:07.817605    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"]
Oct 30 21:46:07.820437 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:07.817860    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp" podUID="cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" containerName="istio-proxy" containerID="cri-o://2e25f9e9a4c36222fc6906178d63732ef6eb73d1852eebbfbfd23d840f9ea989" gracePeriod=30
Oct 30 21:46:11.029128 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:46:11.029027    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:46:13.026389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.026355    5886 generic.go:356] "Generic (PLEG): container finished" podID="cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" containerID="2e25f9e9a4c36222fc6906178d63732ef6eb73d1852eebbfbfd23d840f9ea989" exitCode=0
Oct 30 21:46:13.026827 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.026413    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp" event={"ID":"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1","Type":"ContainerDied","Data":"2e25f9e9a4c36222fc6906178d63732ef6eb73d1852eebbfbfd23d840f9ea989"}
Oct 30 21:46:13.145292 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.145269    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:46:13.212843 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.212807    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-credential-socket\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.213099 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.213079    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-data\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.213304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.213285    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-envoy\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.213433 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.213418    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-certs\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.213573 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.213559    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istiod-ca-cert\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.213683 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.213670    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-kube-api-access-wm5m8\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-kube-api-access-wm5m8\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.213893 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.213878    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-token\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-token\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.214075 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.214061    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-socket\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.214207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.214192    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-podinfo\") pod \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\" (UID: \"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1\") "
Oct 30 21:46:13.217702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.217668    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-podinfo" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:13.217952 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.217932    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-credential-socket" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:13.218141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.218114    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-data" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:13.219611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.219406    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-socket" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:13.219954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.219761    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-certs" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:13.222821 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.220226    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istiod-ca-cert" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:13.222821 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.220404    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-envoy" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:13.222821 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.221495    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-kube-api-access-wm5m8" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "kube-api-access-wm5m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:13.229299 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.229274    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-token" pod "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" (UID: "cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:13.316448 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316410    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:13.316448 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316444    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:13.316639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316459    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:13.316639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316470    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:13.316639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316482    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:13.316639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316521    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm5m8\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-kube-api-access-wm5m8\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:13.316639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316535    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:13.316639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316549    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:13.316639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:13.316561    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:14.036583 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:14.036547    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp" event={"ID":"cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1","Type":"ContainerDied","Data":"a043fbe27e00f0abb672223227e4c0a2cfc9ca692219a40e7e8e9037533ba346"}
Oct 30 21:46:14.036583 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:14.036567    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"
Oct 30 21:46:14.037126 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:14.036614    5886 scope.go:122] "RemoveContainer" containerID="2e25f9e9a4c36222fc6906178d63732ef6eb73d1852eebbfbfd23d840f9ea989"
Oct 30 21:46:14.099023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:14.098986    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"]
Oct 30 21:46:14.110040 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:14.110008    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-0d74b3-openshift-default-8f75fcf5c-qhjfp"]
Oct 30 21:46:15.990707 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:15.990663    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" path="/var/lib/kubelet/pods/cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1/volumes"
Oct 30 21:46:31.682099 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:31.682069    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"]
Oct 30 21:46:31.682550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:31.682279    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy" containerID="cri-o://cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162" gracePeriod=30
Oct 30 21:46:34.222060 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:34.222025    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"]
Oct 30 21:46:34.222492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:34.222226    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy" containerID="cri-o://e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb" gracePeriod=30
Oct 30 21:46:36.981374 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:36.981338    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:46:37.092211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092150    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-data\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.092211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092243    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076366e0-68f8-408b-9867-a5ef93ef1987-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076366e0-68f8-408b-9867-a5ef93ef1987-istiod-ca-cert\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.092550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092301    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-envoy\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.092550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092372    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-socket\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.092550 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092544    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/076366e0-68f8-408b-9867-a5ef93ef1987-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076366e0-68f8-408b-9867-a5ef93ef1987-istio-podinfo\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.092731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092576    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-credential-socket\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.092814 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092783    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-certs\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.092919 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092902    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-kube-api-access-cpptx\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-kube-api-access-cpptx\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.092996 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.092968    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-istio-token\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-istio-token\") pod \"076366e0-68f8-408b-9867-a5ef93ef1987\" (UID: \"076366e0-68f8-408b-9867-a5ef93ef1987\") "
Oct 30 21:46:37.094384 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.094351    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-data" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:37.094786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.094758    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076366e0-68f8-408b-9867-a5ef93ef1987-istiod-ca-cert" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:37.095352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.095324    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-credential-socket" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:37.095535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.095515    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-socket" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:37.095880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.095608    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-certs" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:37.098636 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.098597    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-kube-api-access-cpptx" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "kube-api-access-cpptx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:37.100547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.100518    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/076366e0-68f8-408b-9867-a5ef93ef1987-istio-podinfo" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:37.100785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.100751    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-istio-token" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:37.102647 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.102617    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-envoy" pod "076366e0-68f8-408b-9867-a5ef93ef1987" (UID: "076366e0-68f8-408b-9867-a5ef93ef1987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:37.195800 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195752    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.195800 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195800    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpptx\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-kube-api-access-cpptx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.196034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195818    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076366e0-68f8-408b-9867-a5ef93ef1987-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.196034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195831    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.196034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195844    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076366e0-68f8-408b-9867-a5ef93ef1987-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.196034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195856    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.196034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195870    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.196034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195882    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076366e0-68f8-408b-9867-a5ef93ef1987-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.196034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.195895    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076366e0-68f8-408b-9867-a5ef93ef1987-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:37.223833 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.223732    5886 generic.go:356] "Generic (PLEG): container finished" podID="076366e0-68f8-408b-9867-a5ef93ef1987" containerID="cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162" exitCode=0
Oct 30 21:46:37.223833 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.223800    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"
Oct 30 21:46:37.224078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.223804    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" event={"ID":"076366e0-68f8-408b-9867-a5ef93ef1987","Type":"ContainerDied","Data":"cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162"}
Oct 30 21:46:37.224078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.223923    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4" event={"ID":"076366e0-68f8-408b-9867-a5ef93ef1987","Type":"ContainerDied","Data":"34ddc3c23e6a6032ebd4a3227639fc6eb0a1d8a3a58cc3ece69284ec9a86ba36"}
Oct 30 21:46:37.224078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.223946    5886 scope.go:122] "RemoveContainer" containerID="cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162"
Oct 30 21:46:37.235216 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.235190    5886 scope.go:122] "RemoveContainer" containerID="cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162"
Oct 30 21:46:37.235504 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:46:37.235483    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162\": container with ID starting with cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162 not found: ID does not exist" containerID="cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162"
Oct 30 21:46:37.235569 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.235512    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162"} err="failed to get container status \"cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162\": rpc error: code = NotFound desc = could not find container \"cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162\": container with ID starting with cfeb203e2adbefcf601679ae2209404897b46baac5bb3b7c50cbceb60c9ae162 not found: ID does not exist"
Oct 30 21:46:37.278658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.278580    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"]
Oct 30 21:46:37.284006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.283971    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-82hl4"]
Oct 30 21:46:37.989602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:37.989565    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" path="/var/lib/kubelet/pods/076366e0-68f8-408b-9867-a5ef93ef1987/volumes"
Oct 30 21:46:39.502510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.502448    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:46:39.636477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.636439    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-envoy\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.636660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.636485    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-podinfo\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.636660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.636553    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-token\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-token\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.636660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.636571    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-credential-socket\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.636839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.636711    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c298e40-0c53-4e35-86ab-7d5801546c66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c298e40-0c53-4e35-86ab-7d5801546c66-istiod-ca-cert\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.636839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.636797    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-certs\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.636963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.636847    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-credential-socket" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:39.636963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.636928    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-kube-api-access-cpxt4\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-kube-api-access-cpxt4\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.637050 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637021    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-data\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.637096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637063    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-socket\") pod \"7c298e40-0c53-4e35-86ab-7d5801546c66\" (UID: \"7c298e40-0c53-4e35-86ab-7d5801546c66\") "
Oct 30 21:46:39.637132 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637104    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-certs" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:39.637213 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637143    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c298e40-0c53-4e35-86ab-7d5801546c66-istiod-ca-cert" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:39.637412 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637367    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-data" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:39.637597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637425    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-socket" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:39.637789 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637768    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:39.637839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637798    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c298e40-0c53-4e35-86ab-7d5801546c66-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:39.637839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637815    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:39.637839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637829    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:39.637938 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.637842    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:39.638821 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.638780    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-envoy" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:39.639115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.639079    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-token" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:39.639245 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.639146    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-podinfo" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:39.639384 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.639367    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-kube-api-access-cpxt4" pod "7c298e40-0c53-4e35-86ab-7d5801546c66" (UID: "7c298e40-0c53-4e35-86ab-7d5801546c66"). InnerVolumeSpecName "kube-api-access-cpxt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:39.739230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.739198    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxt4\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-kube-api-access-cpxt4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:39.739230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.739226    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:39.739230 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.739238    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:39.739456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:39.739250    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c298e40-0c53-4e35-86ab-7d5801546c66-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:40.250431 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.250397    5886 generic.go:356] "Generic (PLEG): container finished" podID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerID="e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb" exitCode=0
Oct 30 21:46:40.250603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.250448    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"
Oct 30 21:46:40.250603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.250468    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" event={"ID":"7c298e40-0c53-4e35-86ab-7d5801546c66","Type":"ContainerDied","Data":"e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb"}
Oct 30 21:46:40.250603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.250512    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh" event={"ID":"7c298e40-0c53-4e35-86ab-7d5801546c66","Type":"ContainerDied","Data":"8172a6c0e3c02d1df5c24831299a8681f818dad373b063ed52fddc9d37296a55"}
Oct 30 21:46:40.250603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.250532    5886 scope.go:122] "RemoveContainer" containerID="e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb"
Oct 30 21:46:40.261274 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.261257    5886 scope.go:122] "RemoveContainer" containerID="e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb"
Oct 30 21:46:40.261592 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:46:40.261570    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb\": container with ID starting with e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb not found: ID does not exist" containerID="e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb"
Oct 30 21:46:40.261687 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.261598    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb"} err="failed to get container status \"e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb\": rpc error: code = NotFound desc = could not find container \"e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb\": container with ID starting with e3fba8cb64d3f97c95be19f091c1bd055e30edd70dfa81e886dd2eba58eb52bb not found: ID does not exist"
Oct 30 21:46:40.300691 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.300665    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"]
Oct 30 21:46:40.313235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:40.313209    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-lfjfh"]
Oct 30 21:46:41.031608 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:46:41.031570    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:46:41.988833 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:41.988799    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" path="/var/lib/kubelet/pods/7c298e40-0c53-4e35-86ab-7d5801546c66/volumes"
Oct 30 21:46:44.270602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:44.270567    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"]
Oct 30 21:46:44.270979 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:44.270809    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy" containerID="cri-o://df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e" gracePeriod=30
Oct 30 21:46:49.550199 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.550103    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:46:49.694735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.694702    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-istio-token\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-istio-token\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.694927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.694754    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-certs\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.694927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.694782    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-kube-api-access-pcdsb\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-kube-api-access-pcdsb\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.694927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.694805    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-envoy\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.694927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.694909    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-data\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.695194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.694975    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-credential-socket\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.695194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.695032    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f00c30-1797-45cb-a70e-89188f259de4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f00c30-1797-45cb-a70e-89188f259de4-istiod-ca-cert\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.695194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.695066    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-socket\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.695387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.695233    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8f00c30-1797-45cb-a70e-89188f259de4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f00c30-1797-45cb-a70e-89188f259de4-istio-podinfo\") pod \"a8f00c30-1797-45cb-a70e-89188f259de4\" (UID: \"a8f00c30-1797-45cb-a70e-89188f259de4\") "
Oct 30 21:46:49.695387 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.695369    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-data" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:49.695505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.695438    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-certs" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:49.695505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.695450    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-socket" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:49.695642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.695620    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-credential-socket" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:49.695702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.695659    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f00c30-1797-45cb-a70e-89188f259de4-istiod-ca-cert" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:49.696456 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.696430    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f00c30-1797-45cb-a70e-89188f259de4-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:49.696602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.696459    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:49.696602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.696473    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:49.696602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.696490    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:49.696602 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.696503    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:49.697494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.697465    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-envoy" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:49.697661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.697635    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-istio-token" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:49.698149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.698127    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-kube-api-access-pcdsb" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "kube-api-access-pcdsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:49.698265 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.698212    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8f00c30-1797-45cb-a70e-89188f259de4-istio-podinfo" pod "a8f00c30-1797-45cb-a70e-89188f259de4" (UID: "a8f00c30-1797-45cb-a70e-89188f259de4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:49.798097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.798056    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcdsb\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-kube-api-access-pcdsb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:49.798097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.798094    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f00c30-1797-45cb-a70e-89188f259de4-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:49.798318 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.798107    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f00c30-1797-45cb-a70e-89188f259de4-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:49.798318 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:49.798118    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f00c30-1797-45cb-a70e-89188f259de4-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:46:50.330508 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.330431    5886 generic.go:356] "Generic (PLEG): container finished" podID="a8f00c30-1797-45cb-a70e-89188f259de4" containerID="df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e" exitCode=0
Oct 30 21:46:50.330508 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.330472    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"
Oct 30 21:46:50.330708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.330516    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" event={"ID":"a8f00c30-1797-45cb-a70e-89188f259de4","Type":"ContainerDied","Data":"df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e"}
Oct 30 21:46:50.330708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.330572    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td" event={"ID":"a8f00c30-1797-45cb-a70e-89188f259de4","Type":"ContainerDied","Data":"ed655e9b5837f3c762dcc0b2a361b6e59dc8331af803c5cb7556e289afcd92d3"}
Oct 30 21:46:50.330708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.330595    5886 scope.go:122] "RemoveContainer" containerID="df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e"
Oct 30 21:46:50.341149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.341131    5886 scope.go:122] "RemoveContainer" containerID="df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e"
Oct 30 21:46:50.341492 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:46:50.341470    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e\": container with ID starting with df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e not found: ID does not exist" containerID="df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e"
Oct 30 21:46:50.341557 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.341504    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e"} err="failed to get container status \"df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e\": rpc error: code = NotFound desc = could not find container \"df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e\": container with ID starting with df0a7ce2b0865c6b5b91600077dd1c44ec918d76b8d6745728adf766bb6e5f9e not found: ID does not exist"
Oct 30 21:46:50.425174 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.422233    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"]
Oct 30 21:46:50.431457 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:50.429621    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-dr6td"]
Oct 30 21:46:51.988895 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:51.988863    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" path="/var/lib/kubelet/pods/a8f00c30-1797-45cb-a70e-89188f259de4/volumes"
Oct 30 21:46:58.607263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:58.607229    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"]
Oct 30 21:46:58.607767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:46:58.607418    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" containerID="cri-o://6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6" gracePeriod=30
Oct 30 21:47:02.934446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:02.934407    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:47:02.934866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:02.934496    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:47:03.914404 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.914343    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.961380    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-envoy\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.963757    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-istio-token\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-istio-token\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.964678    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e24781-6814-4429-9f0f-c47017248f06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e24781-6814-4429-9f0f-c47017248f06-istiod-ca-cert\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.964738    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-credential-socket\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.964846    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-data\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.964909    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6e24781-6814-4429-9f0f-c47017248f06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e24781-6814-4429-9f0f-c47017248f06-istio-podinfo\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.965079    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e24781-6814-4429-9f0f-c47017248f06-istiod-ca-cert" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.965111    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-certs\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.965256    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-socket\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.965297    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-kube-api-access-5ffw5\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-kube-api-access-5ffw5\") pod \"d6e24781-6814-4429-9f0f-c47017248f06\" (UID: \"d6e24781-6814-4429-9f0f-c47017248f06\") "
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.965800    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-credential-socket" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.966554    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-socket" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.966673    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-certs" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.967492    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-data" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.967717    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6e24781-6814-4429-9f0f-c47017248f06-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.967735    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.967748    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.967761    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:03.968860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.967775    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:03.978239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:03.975716    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-istio-token" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:04.011060 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.008774    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-envoy" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:04.011060 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.009409    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6e24781-6814-4429-9f0f-c47017248f06-istio-podinfo" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:47:04.011341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.011313    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-kube-api-access-5ffw5" pod "d6e24781-6814-4429-9f0f-c47017248f06" (UID: "d6e24781-6814-4429-9f0f-c47017248f06"). InnerVolumeSpecName "kube-api-access-5ffw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:04.069618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.069582    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6e24781-6814-4429-9f0f-c47017248f06-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:04.069618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.069618    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ffw5\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-kube-api-access-5ffw5\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:04.069859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.069632    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6e24781-6814-4429-9f0f-c47017248f06-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:04.069859 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.069644    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e24781-6814-4429-9f0f-c47017248f06-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:04.482881 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.482842    5886 generic.go:356] "Generic (PLEG): container finished" podID="d6e24781-6814-4429-9f0f-c47017248f06" containerID="6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6" exitCode=0
Oct 30 21:47:04.483089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.482887    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"
Oct 30 21:47:04.483089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.482920    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" event={"ID":"d6e24781-6814-4429-9f0f-c47017248f06","Type":"ContainerDied","Data":"6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6"}
Oct 30 21:47:04.483089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.482976    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq" event={"ID":"d6e24781-6814-4429-9f0f-c47017248f06","Type":"ContainerDied","Data":"280d2c6462accedf9ee9d9a38e642496fc2b15d851c1124675e75f34f9551300"}
Oct 30 21:47:04.483089 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.483002    5886 scope.go:122] "RemoveContainer" containerID="6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6"
Oct 30 21:47:04.495418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.495390    5886 scope.go:122] "RemoveContainer" containerID="6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6"
Oct 30 21:47:04.495744 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:47:04.495712    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6\": container with ID starting with 6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6 not found: ID does not exist" containerID="6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6"
Oct 30 21:47:04.495862 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.495752    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6"} err="failed to get container status \"6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6\": rpc error: code = NotFound desc = could not find container \"6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6\": container with ID starting with 6e486d72394aa52649eaec62e8d24c113b65de62b19dffc20bf37d2dd3d2f0c6 not found: ID does not exist"
Oct 30 21:47:04.566851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.566813    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"]
Oct 30 21:47:04.586093 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:04.586060    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-p69fq"]
Oct 30 21:47:05.991102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:05.991059    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e24781-6814-4429-9f0f-c47017248f06" path="/var/lib/kubelet/pods/d6e24781-6814-4429-9f0f-c47017248f06/volumes"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.481330    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"]
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482311    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482334    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482350    5886 cpu_manager.go:414] "containerMap: removing container" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482358    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482368    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482376    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482392    5886 cpu_manager.go:414] "containerMap: removing container" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482400    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482411    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482418    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482433    5886 cpu_manager.go:414] "containerMap: removing container" podUID="cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482442    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482456    5886 cpu_manager.go:414] "containerMap: removing container" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482463    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482480    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482488    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482550    5886 cpu_manager.go:414] "containerMap: removing container" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.482558    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.496408    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="076366e0-68f8-408b-9867-a5ef93ef1987" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.496447    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e24781-6814-4429-9f0f-c47017248f06" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.496459    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.496469    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f00c30-1797-45cb-a70e-89188f259de4" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.496479    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c298e40-0c53-4e35-86ab-7d5801546c66" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.496487    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9fb635f-90e4-49e9-9a8f-f4e185a8a50e" containerName="cleanup"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.496496    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5b82aa-2eac-4b99-9b14-088a0ff2a7c1" containerName="istio-proxy"
Oct 30 21:47:06.500186 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.496503    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="15707226-8100-4108-98f0-8697986f92af" containerName="istio-proxy"
Oct 30 21:47:06.531221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.504136    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:06.555046 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.554751    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3bc8ea\""
Oct 30 21:47:06.555046 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.554880    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3bc8ea-dockercfg-f9nqc\""
Oct 30 21:47:06.570254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.570219    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"]
Oct 30 21:47:06.625345 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.625309    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87f52dc1-013b-4314-afcf-f0326ec97164-plugins-config-volume\") pod \"payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc\" (UID: \"87f52dc1-013b-4314-afcf-f0326ec97164\") " pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:06.625514 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.625499    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdqj\" (UniqueName: \"kubernetes.io/projected/87f52dc1-013b-4314-afcf-f0326ec97164-kube-api-access-wzdqj\") pod \"payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc\" (UID: \"87f52dc1-013b-4314-afcf-f0326ec97164\") " pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:06.726271 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.726236    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87f52dc1-013b-4314-afcf-f0326ec97164-plugins-config-volume\") pod \"payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc\" (UID: \"87f52dc1-013b-4314-afcf-f0326ec97164\") " pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:06.726477 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.726352    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdqj\" (UniqueName: \"kubernetes.io/projected/87f52dc1-013b-4314-afcf-f0326ec97164-kube-api-access-wzdqj\") pod \"payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc\" (UID: \"87f52dc1-013b-4314-afcf-f0326ec97164\") " pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:06.727002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.726973    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87f52dc1-013b-4314-afcf-f0326ec97164-plugins-config-volume\") pod \"payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc\" (UID: \"87f52dc1-013b-4314-afcf-f0326ec97164\") " pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:06.745382 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.745355    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdqj\" (UniqueName: \"kubernetes.io/projected/87f52dc1-013b-4314-afcf-f0326ec97164-kube-api-access-wzdqj\") pod \"payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc\" (UID: \"87f52dc1-013b-4314-afcf-f0326ec97164\") " pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:06.867515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:06.867427    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:07.182197 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:07.182121    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"]
Oct 30 21:47:07.187203 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:47:07.187135    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f52dc1_013b_4314_afcf_f0326ec97164.slice/crio-992226971fc297d1b480272a69a714c588ecb9073510bc6ea0d7810d25d13221 WatchSource:0}: Error finding container 992226971fc297d1b480272a69a714c588ecb9073510bc6ea0d7810d25d13221: Status 404 returned error can't find the container with id 992226971fc297d1b480272a69a714c588ecb9073510bc6ea0d7810d25d13221
Oct 30 21:47:07.570397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:07.570304    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc" event={"ID":"87f52dc1-013b-4314-afcf-f0326ec97164","Type":"ContainerStarted","Data":"b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451"}
Oct 30 21:47:07.570397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:07.570365    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc" event={"ID":"87f52dc1-013b-4314-afcf-f0326ec97164","Type":"ContainerStarted","Data":"992226971fc297d1b480272a69a714c588ecb9073510bc6ea0d7810d25d13221"}
Oct 30 21:47:07.570630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:07.570519    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:07.611965 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:07.611909    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc" podStartSLOduration=1.61189483 podStartE2EDuration="1.61189483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:07.609459731 +0000 UTC m=+2227.738711190" watchObservedRunningTime="2026-10-30 21:47:07.61189483 +0000 UTC m=+2227.741146286"
Oct 30 21:47:11.030693 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:47:11.030645    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:47:18.580718 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:18.580671    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:47:31.877297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:31.877261    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"]
Oct 30 21:47:31.877845 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:31.877459    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy" containerID="cri-o://98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13" gracePeriod=30
Oct 30 21:47:37.313302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.313248    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:47:37.383822 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.383775    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a4be78-7304-47b5-a11e-11954990b452-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a4be78-7304-47b5-a11e-11954990b452-istiod-ca-cert\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.384042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.383863    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-socket\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.384042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.383936    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19a4be78-7304-47b5-a11e-11954990b452-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a4be78-7304-47b5-a11e-11954990b452-istio-podinfo\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.384042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.383967    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-credential-socket\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.384042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.384024    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-istio-token\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-istio-token\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.384333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.384090    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-envoy\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.384333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.384215    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-certs\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.384333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.384262    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-kube-api-access-jdz6x\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-kube-api-access-jdz6x\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.384506 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.384360    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-data\") pod \"19a4be78-7304-47b5-a11e-11954990b452\" (UID: \"19a4be78-7304-47b5-a11e-11954990b452\") "
Oct 30 21:47:37.385260 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.385231    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-data" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:37.386584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.386558    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a4be78-7304-47b5-a11e-11954990b452-istiod-ca-cert" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:37.388384 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.387916    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-certs" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:37.388384 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.388111    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-credential-socket" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:37.388840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.388815    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-socket" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:37.414393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.414335    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-envoy" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:37.417760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.415256    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19a4be78-7304-47b5-a11e-11954990b452-istio-podinfo" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:47:37.417760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.415381    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-kube-api-access-jdz6x" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "kube-api-access-jdz6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:37.423468 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.423432    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-istio-token" pod "19a4be78-7304-47b5-a11e-11954990b452" (UID: "19a4be78-7304-47b5-a11e-11954990b452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487286    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487320    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19a4be78-7304-47b5-a11e-11954990b452-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487334    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487348    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19a4be78-7304-47b5-a11e-11954990b452-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487362    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487378    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487392    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487405    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19a4be78-7304-47b5-a11e-11954990b452-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.488180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.487418    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdz6x\" (UniqueName: \"kubernetes.io/projected/19a4be78-7304-47b5-a11e-11954990b452-kube-api-access-jdz6x\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:47:37.887939 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.887901    5886 generic.go:356] "Generic (PLEG): container finished" podID="19a4be78-7304-47b5-a11e-11954990b452" containerID="98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13" exitCode=0
Oct 30 21:47:37.888128 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.887950    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"
Oct 30 21:47:37.888128 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.887978    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" event={"ID":"19a4be78-7304-47b5-a11e-11954990b452","Type":"ContainerDied","Data":"98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13"}
Oct 30 21:47:37.888128 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.888075    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn" event={"ID":"19a4be78-7304-47b5-a11e-11954990b452","Type":"ContainerDied","Data":"a2f117307e025e72adf07a01dbb7836a8a77ea2e794fefd4676f6df45cb8ef8c"}
Oct 30 21:47:37.888128 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.888105    5886 scope.go:122] "RemoveContainer" containerID="98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13"
Oct 30 21:47:37.899849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.899822    5886 scope.go:122] "RemoveContainer" containerID="98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13"
Oct 30 21:47:37.900186 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:47:37.900141    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13\": container with ID starting with 98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13 not found: ID does not exist" containerID="98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13"
Oct 30 21:47:37.900321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.900195    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13"} err="failed to get container status \"98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13\": rpc error: code = NotFound desc = could not find container \"98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13\": container with ID starting with 98ac67018fef3d54599e102a4becb02086525edd0bef9cd1c622412107639b13 not found: ID does not exist"
Oct 30 21:47:37.964017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.963986    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"]
Oct 30 21:47:37.976482 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.976455    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vgpcn"]
Oct 30 21:47:37.988815 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:37.988792    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a4be78-7304-47b5-a11e-11954990b452" path="/var/lib/kubelet/pods/19a4be78-7304-47b5-a11e-11954990b452/volumes"
Oct 30 21:47:41.030570 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:47:41.030489    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:47:41.669338 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.669300    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"]
Oct 30 21:47:41.669966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.669948    5886 cpu_manager.go:414] "containerMap: removing container" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy"
Oct 30 21:47:41.669966 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.669965    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy"
Oct 30 21:47:41.670127 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.670052    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8301798-2880-481f-ad6d-189c09446234" containerName="istio-proxy"
Oct 30 21:47:41.670127 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.670070    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a4be78-7304-47b5-a11e-11954990b452" containerName="istio-proxy"
Oct 30 21:47:41.675492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.675393    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.700344 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.700317    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-427a6e-serving-cert\""
Oct 30 21:47:41.712068 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.711564    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"]
Oct 30 21:47:41.749004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.744566    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-metrics-certs\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.749004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.744662    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.749004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.744802    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xvnx\" (UniqueName: \"kubernetes.io/projected/fff50fba-fab4-498f-bd01-8613f3e02a9d-kube-api-access-2xvnx\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.749004 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.744890    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff50fba-fab4-498f-bd01-8613f3e02a9d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.845799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.845758    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.846018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.845907    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xvnx\" (UniqueName: \"kubernetes.io/projected/fff50fba-fab4-498f-bd01-8613f3e02a9d-kube-api-access-2xvnx\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.846018 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.845980    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff50fba-fab4-498f-bd01-8613f3e02a9d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.846126 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.846068    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-metrics-certs\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.846612 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.846585    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff50fba-fab4-498f-bd01-8613f3e02a9d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.848611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.848587    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.848828 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.848800    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-metrics-certs\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.869134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.869106    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xvnx\" (UniqueName: \"kubernetes.io/projected/fff50fba-fab4-498f-bd01-8613f3e02a9d-kube-api-access-2xvnx\") pod \"maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:41.995997 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:41.995910    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:42.306643 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:42.306568    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"]
Oct 30 21:47:42.310056 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:47:42.310027    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff50fba_fab4_498f_bd01_8613f3e02a9d.slice/crio-fbb90ebfcf4c89d7f3e9bbc6cf8b928a634280f06842605a18aa156fcb5aefb4 WatchSource:0}: Error finding container fbb90ebfcf4c89d7f3e9bbc6cf8b928a634280f06842605a18aa156fcb5aefb4: Status 404 returned error can't find the container with id fbb90ebfcf4c89d7f3e9bbc6cf8b928a634280f06842605a18aa156fcb5aefb4
Oct 30 21:47:42.928797 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:42.928765    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c" event={"ID":"fff50fba-fab4-498f-bd01-8613f3e02a9d","Type":"ContainerStarted","Data":"2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886"}
Oct 30 21:47:42.929009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:42.928814    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c" event={"ID":"fff50fba-fab4-498f-bd01-8613f3e02a9d","Type":"ContainerStarted","Data":"fbb90ebfcf4c89d7f3e9bbc6cf8b928a634280f06842605a18aa156fcb5aefb4"}
Oct 30 21:47:42.929009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:42.928963    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:42.978491 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:42.978430    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c" podStartSLOduration=1.6631539549999999 podStartE2EDuration="1.978416419s" totalImagesPullingTime="315.262464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:42.97453211 +0000 UTC m=+2263.103783568" watchObservedRunningTime="2026-10-30 21:47:42.978416419 +0000 UTC m=+2263.107667875"
Oct 30 21:47:43.922217 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:43.922182    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"]
Oct 30 21:47:43.926625 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:43.926605    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:43.961280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:43.961253    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-846656\""
Oct 30 21:47:43.961280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:43.961253    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-846656-dockercfg-j725h\""
Oct 30 21:47:43.965676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:43.965637    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"]
Oct 30 21:47:43.983873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:43.982958    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrgf\" (UniqueName: \"kubernetes.io/projected/98aee083-5553-48d7-a3d9-451c5588e455-kube-api-access-gqrgf\") pod \"payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md\" (UID: \"98aee083-5553-48d7-a3d9-451c5588e455\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:43.983873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:43.983128    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98aee083-5553-48d7-a3d9-451c5588e455-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md\" (UID: \"98aee083-5553-48d7-a3d9-451c5588e455\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:44.102475 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.102209    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"]
Oct 30 21:47:44.118319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.111522    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98aee083-5553-48d7-a3d9-451c5588e455-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md\" (UID: \"98aee083-5553-48d7-a3d9-451c5588e455\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:44.118319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.111735    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrgf\" (UniqueName: \"kubernetes.io/projected/98aee083-5553-48d7-a3d9-451c5588e455-kube-api-access-gqrgf\") pod \"payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md\" (UID: \"98aee083-5553-48d7-a3d9-451c5588e455\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:44.118319 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.117499    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98aee083-5553-48d7-a3d9-451c5588e455-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md\" (UID: \"98aee083-5553-48d7-a3d9-451c5588e455\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:44.121254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.121226    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:44.174200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.171932    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrgf\" (UniqueName: \"kubernetes.io/projected/98aee083-5553-48d7-a3d9-451c5588e455-kube-api-access-gqrgf\") pod \"payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md\" (UID: \"98aee083-5553-48d7-a3d9-451c5588e455\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:44.174200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.174055    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"]
Oct 30 21:47:44.213356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.213325    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53d2c15f-71ee-4335-bf9d-50632aab2c62-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd\" (UID: \"53d2c15f-71ee-4335-bf9d-50632aab2c62\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:44.213548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.213494    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fzg\" (UniqueName: \"kubernetes.io/projected/53d2c15f-71ee-4335-bf9d-50632aab2c62-kube-api-access-t2fzg\") pod \"payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd\" (UID: \"53d2c15f-71ee-4335-bf9d-50632aab2c62\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:44.242735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.242703    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:44.314583 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.314555    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53d2c15f-71ee-4335-bf9d-50632aab2c62-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd\" (UID: \"53d2c15f-71ee-4335-bf9d-50632aab2c62\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:44.314720 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.314647    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fzg\" (UniqueName: \"kubernetes.io/projected/53d2c15f-71ee-4335-bf9d-50632aab2c62-kube-api-access-t2fzg\") pod \"payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd\" (UID: \"53d2c15f-71ee-4335-bf9d-50632aab2c62\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:44.315358 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.315337    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53d2c15f-71ee-4335-bf9d-50632aab2c62-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd\" (UID: \"53d2c15f-71ee-4335-bf9d-50632aab2c62\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:44.340302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.340251    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fzg\" (UniqueName: \"kubernetes.io/projected/53d2c15f-71ee-4335-bf9d-50632aab2c62-kube-api-access-t2fzg\") pod \"payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd\" (UID: \"53d2c15f-71ee-4335-bf9d-50632aab2c62\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:44.439030 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.438941    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:44.593989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.593952    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"]
Oct 30 21:47:44.599246 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:47:44.599213    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98aee083_5553_48d7_a3d9_451c5588e455.slice/crio-3b94006b24b0fa818b304d1ea41db74b90a99b806e6f9b5672748ace7f42a418 WatchSource:0}: Error finding container 3b94006b24b0fa818b304d1ea41db74b90a99b806e6f9b5672748ace7f42a418: Status 404 returned error can't find the container with id 3b94006b24b0fa818b304d1ea41db74b90a99b806e6f9b5672748ace7f42a418
Oct 30 21:47:44.777699 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.777661    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"]
Oct 30 21:47:44.780651 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:47:44.780620    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d2c15f_71ee_4335_bf9d_50632aab2c62.slice/crio-eb922a4ef731a25ce2490f3a34c1a93e0d5870f8bf95cdeb3a96c1872c0eb1b0 WatchSource:0}: Error finding container eb922a4ef731a25ce2490f3a34c1a93e0d5870f8bf95cdeb3a96c1872c0eb1b0: Status 404 returned error can't find the container with id eb922a4ef731a25ce2490f3a34c1a93e0d5870f8bf95cdeb3a96c1872c0eb1b0
Oct 30 21:47:44.947818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.947730    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md" event={"ID":"98aee083-5553-48d7-a3d9-451c5588e455","Type":"ContainerStarted","Data":"4729909e56ca9227467acab15e176c75a346dc2dae1832e2e9ff21b298a91a9b"}
Oct 30 21:47:44.947818 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.947811    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md" event={"ID":"98aee083-5553-48d7-a3d9-451c5588e455","Type":"ContainerStarted","Data":"3b94006b24b0fa818b304d1ea41db74b90a99b806e6f9b5672748ace7f42a418"}
Oct 30 21:47:44.948343 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.947943    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:47:44.949483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.949453    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd" event={"ID":"53d2c15f-71ee-4335-bf9d-50632aab2c62","Type":"ContainerStarted","Data":"af87f71271b08ba18228580e6ea68b928d811e8cf6eb2d36d14ed91d212c3105"}
Oct 30 21:47:44.949586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.949506    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd" event={"ID":"53d2c15f-71ee-4335-bf9d-50632aab2c62","Type":"ContainerStarted","Data":"eb922a4ef731a25ce2490f3a34c1a93e0d5870f8bf95cdeb3a96c1872c0eb1b0"}
Oct 30 21:47:44.949586 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:44.949539    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:45.002280 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:45.002215    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md" podStartSLOduration=2.002203233 podStartE2EDuration="2.002203233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:45.000902708 +0000 UTC m=+2265.130154185" watchObservedRunningTime="2026-10-30 21:47:45.002203233 +0000 UTC m=+2265.131454689"
Oct 30 21:47:45.059483 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:45.059434    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd" podStartSLOduration=1.059422136 podStartE2EDuration="1.059422136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:45.059023423 +0000 UTC m=+2265.188274882" watchObservedRunningTime="2026-10-30 21:47:45.059422136 +0000 UTC m=+2265.188673592"
Oct 30 21:47:48.942689 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:48.942652    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 21:47:49.109980 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.109940    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"]
Oct 30 21:47:49.114436 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.114415    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:49.136932 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.136881    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-805edc\""
Oct 30 21:47:49.137052 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.137013    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-805edc-dockercfg-5gtlq\""
Oct 30 21:47:49.145324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.145299    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"]
Oct 30 21:47:49.181541 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.181502    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glq4g\" (UniqueName: \"kubernetes.io/projected/c2ceb265-b14e-4cbc-a8a4-9ea026496943-kube-api-access-glq4g\") pod \"payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k\" (UID: \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:49.181844 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.181823    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ceb265-b14e-4cbc-a8a4-9ea026496943-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k\" (UID: \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:49.291224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.285088    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ceb265-b14e-4cbc-a8a4-9ea026496943-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k\" (UID: \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:49.291224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.285186    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glq4g\" (UniqueName: \"kubernetes.io/projected/c2ceb265-b14e-4cbc-a8a4-9ea026496943-kube-api-access-glq4g\") pod \"payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k\" (UID: \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:49.291224 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.286581    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ceb265-b14e-4cbc-a8a4-9ea026496943-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k\" (UID: \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:49.327944 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.327907    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glq4g\" (UniqueName: \"kubernetes.io/projected/c2ceb265-b14e-4cbc-a8a4-9ea026496943-kube-api-access-glq4g\") pod \"payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k\" (UID: \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:49.428510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.428475    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:49.745363 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.745331    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"]
Oct 30 21:47:49.748507 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:47:49.748483    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ceb265_b14e_4cbc_a8a4_9ea026496943.slice/crio-549fe5a76e6f4ae28210e1da8b680ff797f8a3db3c069c12d8ac100a09158fd6 WatchSource:0}: Error finding container 549fe5a76e6f4ae28210e1da8b680ff797f8a3db3c069c12d8ac100a09158fd6: Status 404 returned error can't find the container with id 549fe5a76e6f4ae28210e1da8b680ff797f8a3db3c069c12d8ac100a09158fd6
Oct 30 21:47:49.990558 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.990524    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k" event={"ID":"c2ceb265-b14e-4cbc-a8a4-9ea026496943","Type":"ContainerStarted","Data":"7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c"}
Oct 30 21:47:49.991033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.990570    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k" event={"ID":"c2ceb265-b14e-4cbc-a8a4-9ea026496943","Type":"ContainerStarted","Data":"549fe5a76e6f4ae28210e1da8b680ff797f8a3db3c069c12d8ac100a09158fd6"}
Oct 30 21:47:49.991033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:49.990623    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:47:50.034133 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:50.034022    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k" podStartSLOduration=1.034008628 podStartE2EDuration="1.034008628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:50.033009737 +0000 UTC m=+2270.162261194" watchObservedRunningTime="2026-10-30 21:47:50.034008628 +0000 UTC m=+2270.163260085"
Oct 30 21:47:55.960879 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:55.960841    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 21:47:55.961325 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:47:55.960950    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 21:48:01.003313 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:01.003280    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 21:48:11.028557 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:48:11.028517    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:48:22.030932 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.030893    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"]
Oct 30 21:48:22.036858 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.036825    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.054369 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.054340    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-805edc-serving-cert\""
Oct 30 21:48:22.065936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.065908    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"]
Oct 30 21:48:22.069302 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.069268    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e625489f-694d-4ca8-95f6-3192e3e084e4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.069739 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.069716    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-metrics-certs\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.069931 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.069908    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.070033 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.070010    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vff4q\" (UniqueName: \"kubernetes.io/projected/e625489f-694d-4ca8-95f6-3192e3e084e4-kube-api-access-vff4q\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.171911 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.171874    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.172109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.171945    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vff4q\" (UniqueName: \"kubernetes.io/projected/e625489f-694d-4ca8-95f6-3192e3e084e4-kube-api-access-vff4q\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.172109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.172031    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e625489f-694d-4ca8-95f6-3192e3e084e4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.172109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.172090    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-metrics-certs\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.172825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.172796    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e625489f-694d-4ca8-95f6-3192e3e084e4-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.174385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.174360    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-maas-api-tls\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.174495 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.174391    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-metrics-certs\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.190569 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.190539    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vff4q\" (UniqueName: \"kubernetes.io/projected/e625489f-694d-4ca8-95f6-3192e3e084e4-kube-api-access-vff4q\") pod \"maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.355898 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.355817    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:22.671552 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.671522    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"]
Oct 30 21:48:22.675013 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:48:22.674985    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode625489f_694d_4ca8_95f6_3192e3e084e4.slice/crio-21519bf0d023d24be2f454aaea98de153864a460d855d57b041af68a7c57c6c5 WatchSource:0}: Error finding container 21519bf0d023d24be2f454aaea98de153864a460d855d57b041af68a7c57c6c5: Status 404 returned error can't find the container with id 21519bf0d023d24be2f454aaea98de153864a460d855d57b041af68a7c57c6c5
Oct 30 21:48:22.804910 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.803859    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"]
Oct 30 21:48:22.813138 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.810275    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.825078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.825042    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-db1288-openshift-default-dockercfg-cbbqv\""
Oct 30 21:48:22.842473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.842444    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"]
Oct 30 21:48:22.880494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.880464    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddv8\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-kube-api-access-vddv8\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.880668 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.880643    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-credential-socket\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.880736 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.880705    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-token\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.880908 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.880877    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-certs\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.880963 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.880924    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-data\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.881104 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.881084    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42d676f-8ffd-4663-8771-ec3c17bff079-istiod-ca-cert\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.881191 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.881132    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-podinfo\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.881309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.881288    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-socket\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.881382 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.881363    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-envoy\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992034    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-credential-socket\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992136 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992111    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-token\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992183    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-certs\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992214    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-data\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992306    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42d676f-8ffd-4663-8771-ec3c17bff079-istiod-ca-cert\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992345    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-podinfo\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992418 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992417    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-socket\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992452    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-envoy\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992510    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddv8\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-kube-api-access-vddv8\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.992698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.992536    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-credential-socket\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.994179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.994129    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42d676f-8ffd-4663-8771-ec3c17bff079-istiod-ca-cert\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.994422 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.994398    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-data\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.994648 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.994622    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-certs\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.994964 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.994929    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-socket\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:22.997096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:22.997046    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-podinfo\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:23.001849 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.001811    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-envoy\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:23.014733 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.014702    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddv8\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-kube-api-access-vddv8\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:23.016651 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.016614    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-token\") pod \"e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") " pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:23.127820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.127782    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:23.274472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.273792    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b" event={"ID":"e625489f-694d-4ca8-95f6-3192e3e084e4","Type":"ContainerStarted","Data":"700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421"}
Oct 30 21:48:23.274472 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.273898    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b" event={"ID":"e625489f-694d-4ca8-95f6-3192e3e084e4","Type":"ContainerStarted","Data":"21519bf0d023d24be2f454aaea98de153864a460d855d57b041af68a7c57c6c5"}
Oct 30 21:48:23.274989 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.274914    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:23.471431 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.471360    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b" podStartSLOduration=1.009485894 podStartE2EDuration="1.471341872s" totalImagesPullingTime="461.855978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:23.32382063 +0000 UTC m=+2303.453072109" watchObservedRunningTime="2026-10-30 21:48:23.471341872 +0000 UTC m=+2303.600593344"
Oct 30 21:48:23.477801 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.477771    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"]
Oct 30 21:48:23.487956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.486258    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:23.487956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.486337    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:23.487956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:23.486371    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:48:24.282542 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:24.282503    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" event={"ID":"c42d676f-8ffd-4663-8771-ec3c17bff079","Type":"ContainerStarted","Data":"be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db"}
Oct 30 21:48:24.282985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:24.282567    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" event={"ID":"c42d676f-8ffd-4663-8771-ec3c17bff079","Type":"ContainerStarted","Data":"1390a69ec9574a97f0738ab954cc4e7da98ce6655ad560a99526b8642734761e"}
Oct 30 21:48:24.351517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:24.351429    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" podStartSLOduration=2.351415204 podStartE2EDuration="2.351415204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:24.337123638 +0000 UTC m=+2304.466375105" watchObservedRunningTime="2026-10-30 21:48:24.351415204 +0000 UTC m=+2304.480666661"
Oct 30 21:48:25.128111 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:25.128079    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:25.133782 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:25.133754    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:25.291600 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:25.291570    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:25.292486 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:25.292466    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:48:30.297587 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:30.297554    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 21:48:41.029018 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:48:41.028977    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:48:55.220518 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:48:55.220485    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"]
Oct 30 21:49:00.221266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.221204    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc" podUID="87f52dc1-013b-4314-afcf-f0326ec97164" containerName="payload-processing" containerID="cri-o://b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451" gracePeriod=25
Oct 30 21:49:00.649726 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.649692    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:49:00.665380 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.665341    5886 generic.go:356] "Generic (PLEG): container finished" podID="87f52dc1-013b-4314-afcf-f0326ec97164" containerID="b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451" exitCode=0
Oct 30 21:49:00.665762 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.665740    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc" event={"ID":"87f52dc1-013b-4314-afcf-f0326ec97164","Type":"ContainerDied","Data":"b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451"}
Oct 30 21:49:00.665933 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.665920    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc" event={"ID":"87f52dc1-013b-4314-afcf-f0326ec97164","Type":"ContainerDied","Data":"992226971fc297d1b480272a69a714c588ecb9073510bc6ea0d7810d25d13221"}
Oct 30 21:49:00.666038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.666026    5886 scope.go:122] "RemoveContainer" containerID="b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451"
Oct 30 21:49:00.755388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.750545    5886 scope.go:122] "RemoveContainer" containerID="b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451"
Oct 30 21:49:00.755388 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:49:00.750988    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451\": container with ID starting with b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451 not found: ID does not exist" containerID="b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451"
Oct 30 21:49:00.755388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.751025    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451"} err="failed to get container status \"b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451\": rpc error: code = NotFound desc = could not find container \"b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451\": container with ID starting with b69be061f21589dcd85fd24e2bbe5fa5e011232b2958a5ece60c52facfed9451 not found: ID does not exist"
Oct 30 21:49:00.766180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.760523    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f52dc1-013b-4314-afcf-f0326ec97164-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87f52dc1-013b-4314-afcf-f0326ec97164-plugins-config-volume\") pod \"87f52dc1-013b-4314-afcf-f0326ec97164\" (UID: \"87f52dc1-013b-4314-afcf-f0326ec97164\") "
Oct 30 21:49:00.766180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.760697    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f52dc1-013b-4314-afcf-f0326ec97164-kube-api-access-wzdqj\" (UniqueName: \"kubernetes.io/projected/87f52dc1-013b-4314-afcf-f0326ec97164-kube-api-access-wzdqj\") pod \"87f52dc1-013b-4314-afcf-f0326ec97164\" (UID: \"87f52dc1-013b-4314-afcf-f0326ec97164\") "
Oct 30 21:49:00.766180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.761972    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f52dc1-013b-4314-afcf-f0326ec97164-plugins-config-volume" pod "87f52dc1-013b-4314-afcf-f0326ec97164" (UID: "87f52dc1-013b-4314-afcf-f0326ec97164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:00.781771 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.780826    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f52dc1-013b-4314-afcf-f0326ec97164-kube-api-access-wzdqj" pod "87f52dc1-013b-4314-afcf-f0326ec97164" (UID: "87f52dc1-013b-4314-afcf-f0326ec97164"). InnerVolumeSpecName "kube-api-access-wzdqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:00.863222 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.862500    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzdqj\" (UniqueName: \"kubernetes.io/projected/87f52dc1-013b-4314-afcf-f0326ec97164-kube-api-access-wzdqj\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:00.863222 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:00.862535    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87f52dc1-013b-4314-afcf-f0326ec97164-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:01.710190 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:01.707944    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"
Oct 30 21:49:01.789533 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:01.789502    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"]
Oct 30 21:49:01.796257 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:01.796218    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3bc8ea-84d5bb7bc7-5lkgc"]
Oct 30 21:49:02.001896 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:02.001779    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f52dc1-013b-4314-afcf-f0326ec97164" path="/var/lib/kubelet/pods/87f52dc1-013b-4314-afcf-f0326ec97164/volumes"
Oct 30 21:49:04.127109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.127075    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"]
Oct 30 21:49:04.127731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.127716    5886 cpu_manager.go:414] "containerMap: removing container" podUID="87f52dc1-013b-4314-afcf-f0326ec97164" containerName="payload-processing"
Oct 30 21:49:04.127788 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.127732    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f52dc1-013b-4314-afcf-f0326ec97164" containerName="payload-processing"
Oct 30 21:49:04.127860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.127855    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f52dc1-013b-4314-afcf-f0326ec97164" containerName="payload-processing"
Oct 30 21:49:04.132795 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.132770    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.157590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.157568    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-db1288-serving-cert\""
Oct 30 21:49:04.163703 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.163675    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"]
Oct 30 21:49:04.227242 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.227210    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-metrics-certs\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.227242 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.227247    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6871c7-ebe8-457e-b8af-e2025076b1a3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.227476 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.227454    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk5qq\" (UniqueName: \"kubernetes.io/projected/5f6871c7-ebe8-457e-b8af-e2025076b1a3-kube-api-access-mk5qq\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.227516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.227494    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.333185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.332321    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk5qq\" (UniqueName: \"kubernetes.io/projected/5f6871c7-ebe8-457e-b8af-e2025076b1a3-kube-api-access-mk5qq\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.333185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.332379    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.333185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.332526    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-metrics-certs\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.333185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.332547    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6871c7-ebe8-457e-b8af-e2025076b1a3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.333551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.333314    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6871c7-ebe8-457e-b8af-e2025076b1a3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.334184 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:49:04.333728    5886 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-serving-cert: secret "maas-api-e2e-extmod-b-db1288-serving-cert" not found
Oct 30 21:49:04.334184 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:49:04.333791    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls podName:5f6871c7-ebe8-457e-b8af-e2025076b1a3 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:04.833773315 +0000 UTC m=+2344.963024767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls") pod "maas-api-e2e-extmod-b-db1288-dff644984-v9zws" (UID: "5f6871c7-ebe8-457e-b8af-e2025076b1a3") : secret "maas-api-e2e-extmod-b-db1288-serving-cert" not found
Oct 30 21:49:04.337928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.337899    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-metrics-certs\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.355597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.355572    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk5qq\" (UniqueName: \"kubernetes.io/projected/5f6871c7-ebe8-457e-b8af-e2025076b1a3-kube-api-access-mk5qq\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.838907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.838869    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:04.841309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:04.841282    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-db1288-dff644984-v9zws\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:05.047574 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:05.047546    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:05.376834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:05.376801    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"]
Oct 30 21:49:05.378273 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:49:05.378239    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6871c7_ebe8_457e_b8af_e2025076b1a3.slice/crio-a2fd1eab9e11937287b88f7c044ca25ff3806d97b4fb3a7a6de89a01bd286bcf WatchSource:0}: Error finding container a2fd1eab9e11937287b88f7c044ca25ff3806d97b4fb3a7a6de89a01bd286bcf: Status 404 returned error can't find the container with id a2fd1eab9e11937287b88f7c044ca25ff3806d97b4fb3a7a6de89a01bd286bcf
Oct 30 21:49:05.738576 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:05.738537    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws" event={"ID":"5f6871c7-ebe8-457e-b8af-e2025076b1a3","Type":"ContainerStarted","Data":"a2fd1eab9e11937287b88f7c044ca25ff3806d97b4fb3a7a6de89a01bd286bcf"}
Oct 30 21:49:06.747314 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:06.747281    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws" event={"ID":"5f6871c7-ebe8-457e-b8af-e2025076b1a3","Type":"ContainerStarted","Data":"bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac"}
Oct 30 21:49:06.747696 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:06.747375    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:06.788747 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:06.788689    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws" podStartSLOduration=2.4313020610000002 podStartE2EDuration="2.78867587s" totalImagesPullingTime="357.373809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:06.786852166 +0000 UTC m=+2346.916103624" watchObservedRunningTime="2026-10-30 21:49:06.78867587 +0000 UTC m=+2346.917927327"
Oct 30 21:49:11.030101 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:49:11.030012    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:49:12.760125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:12.760093    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 21:49:28.727872 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:28.727833    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"]
Oct 30 21:49:28.728388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:28.728028    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy" containerID="cri-o://e6b0131ad01eb682c5914eefdea2e728432a5ad7c42ffc8451e7c8ae5821678f" gracePeriod=30
Oct 30 21:49:33.983734 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:33.983696    5886 generic.go:356] "Generic (PLEG): container finished" podID="1d73009a-9704-4c93-80fa-df3da2f71609" containerID="e6b0131ad01eb682c5914eefdea2e728432a5ad7c42ffc8451e7c8ae5821678f" exitCode=0
Oct 30 21:49:33.993650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:33.993619    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" event={"ID":"1d73009a-9704-4c93-80fa-df3da2f71609","Type":"ContainerDied","Data":"e6b0131ad01eb682c5914eefdea2e728432a5ad7c42ffc8451e7c8ae5821678f"}
Oct 30 21:49:34.028906 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.028881    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:49:34.164304 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164256    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-data\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.164536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164341    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-envoy\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.164536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164396    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-istio-token\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-istio-token\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.164536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164461    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d73009a-9704-4c93-80fa-df3da2f71609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d73009a-9704-4c93-80fa-df3da2f71609-istio-podinfo\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.164536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164507    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-certs\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.164764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164548    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-credential-socket\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.164764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164605    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-kube-api-access-9vb4b\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-kube-api-access-9vb4b\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.164764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164639    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-socket\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.164764 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.164681    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d73009a-9704-4c93-80fa-df3da2f71609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d73009a-9704-4c93-80fa-df3da2f71609-istiod-ca-cert\") pod \"1d73009a-9704-4c93-80fa-df3da2f71609\" (UID: \"1d73009a-9704-4c93-80fa-df3da2f71609\") "
Oct 30 21:49:34.165856 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.165820    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-data" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.165992 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.165866    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-socket" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.166071 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.166047    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-certs" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.166454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.166227    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-credential-socket" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.166454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.166401    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d73009a-9704-4c93-80fa-df3da2f71609-istiod-ca-cert" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:34.168286 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.168254    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d73009a-9704-4c93-80fa-df3da2f71609-istio-podinfo" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:49:34.168392 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.168336    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-envoy" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.174646 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.174611    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-istio-token" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:34.174814 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.174681    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-kube-api-access-9vb4b" pod "1d73009a-9704-4c93-80fa-df3da2f71609" (UID: "1d73009a-9704-4c93-80fa-df3da2f71609"). InnerVolumeSpecName "kube-api-access-9vb4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:34.266798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266720    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.266798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266750    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.266798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266759    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.266798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266767    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d73009a-9704-4c93-80fa-df3da2f71609-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.266798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266776    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.266798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266785    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.266798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266794    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vb4b\" (UniqueName: \"kubernetes.io/projected/1d73009a-9704-4c93-80fa-df3da2f71609-kube-api-access-9vb4b\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.266798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266803    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d73009a-9704-4c93-80fa-df3da2f71609-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.267181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.266815    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d73009a-9704-4c93-80fa-df3da2f71609-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:49:34.993175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.992722    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"
Oct 30 21:49:34.993175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.992749    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv" event={"ID":"1d73009a-9704-4c93-80fa-df3da2f71609","Type":"ContainerDied","Data":"cee59950025089428ffd85e254b6e798d1ccfea7d943a43371ed8190ae4b4ef9"}
Oct 30 21:49:34.993175 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:34.992815    5886 scope.go:122] "RemoveContainer" containerID="e6b0131ad01eb682c5914eefdea2e728432a5ad7c42ffc8451e7c8ae5821678f"
Oct 30 21:49:35.056405 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:35.056373    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"]
Oct 30 21:49:35.068749 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:35.068718    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-4p8lv"]
Oct 30 21:49:35.990668 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:35.990636    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" path="/var/lib/kubelet/pods/1d73009a-9704-4c93-80fa-df3da2f71609/volumes"
Oct 30 21:49:38.088180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.087470    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"]
Oct 30 21:49:38.088702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.088483    5886 cpu_manager.go:414] "containerMap: removing container" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy"
Oct 30 21:49:38.088702 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.088509    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy"
Oct 30 21:49:38.088820 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.088715    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d73009a-9704-4c93-80fa-df3da2f71609" containerName="istio-proxy"
Oct 30 21:49:38.096105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.096074    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"]
Oct 30 21:49:38.096305 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.096223    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.121710 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.121687    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"routing-overlay\""
Oct 30 21:49:38.121903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.121707    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"kube-root-ca.crt\""
Oct 30 21:49:38.122043 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.122026    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"agc-cred-e2da5504c633\""
Oct 30 21:49:38.122119 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.122097    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"openshift-service-ca.crt\""
Oct 30 21:49:38.122200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.122135    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdb168\""
Oct 30 21:49:38.122307 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.122288    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdb168\"/\"payload-processing-external-model-e2e--ca9775af-dockercfg-772nl\""
Oct 30 21:49:38.209134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.209027    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.226183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.222238    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpxg\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-kube-api-access-qnpxg\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.226183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.222501    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.226183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.222614    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.324178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.324017    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.324178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.324116    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpxg\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-kube-api-access-qnpxg\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.324450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.324232    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.324450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.324315    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.331172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.325898    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.331172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.327094    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.335738 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.335683    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.352211 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.351816    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpxg\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-kube-api-access-qnpxg\") pod \"payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") " pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.416678 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.416641    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:38.791307 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:38.791279    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"]
Oct 30 21:49:38.793423 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:49:38.793392    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cb2c67_612a_455e_ab73_1469dc5b4097.slice/crio-9e82f25db614bf3b654dc56526bd36fd539df6379f38360b6b49a97436885efc WatchSource:0}: Error finding container 9e82f25db614bf3b654dc56526bd36fd539df6379f38360b6b49a97436885efc: Status 404 returned error can't find the container with id 9e82f25db614bf3b654dc56526bd36fd539df6379f38360b6b49a97436885efc
Oct 30 21:49:39.025630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:39.025595    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr" event={"ID":"99cb2c67-612a-455e-ab73-1469dc5b4097","Type":"ContainerStarted","Data":"677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8"}
Oct 30 21:49:39.025830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:39.025654    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr" event={"ID":"99cb2c67-612a-455e-ab73-1469dc5b4097","Type":"ContainerStarted","Data":"9e82f25db614bf3b654dc56526bd36fd539df6379f38360b6b49a97436885efc"}
Oct 30 21:49:39.025830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:39.025700    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:39.067232 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:39.066776    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr" podStartSLOduration=1.0667559739999999 podStartE2EDuration="1.066755974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.066410329 +0000 UTC m=+2379.195661788" watchObservedRunningTime="2026-10-30 21:49:39.066755974 +0000 UTC m=+2379.196007431"
Oct 30 21:49:41.029864 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:49:41.029827    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:49:47.094019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:47.093919    5886 generic.go:356] "Generic (PLEG): container finished" podID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerID="be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db" exitCode=0
Oct 30 21:49:47.094019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:47.093996    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" event={"ID":"c42d676f-8ffd-4663-8771-ec3c17bff079","Type":"ContainerDied","Data":"be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db"}
Oct 30 21:49:47.094593 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:47.094470    5886 scope.go:122] "RemoveContainer" containerID="be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db"
Oct 30 21:49:47.096298 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:47.096270    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:47.096424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:47.096337    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:47.096424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:47.096367    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:49:48.106231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:48.106191    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" event={"ID":"c42d676f-8ffd-4663-8771-ec3c17bff079","Type":"ContainerStarted","Data":"8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196"}
Oct 30 21:49:48.128467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:48.128436    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:49:48.128652 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:48.128632    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:49:48.128700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:48.128676    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 21:49:49.128254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:49.128219    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:49:49.128657 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:49.128266    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 21:49:50.054848 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:50.054486    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:49:50.128813 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:50.128782    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:49:50.129253 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:50.128825    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 21:49:51.128510 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:51.128472    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:49:51.128691 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:51.128540    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 21:49:52.128572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:52.128534    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:49:52.128977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:52.128590    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 21:49:53.134481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:53.130531    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:49:53.134481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:53.130592    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 30 21:49:54.134487 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:54.134453    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:49:54.134891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:54.134841    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:49:54.135760 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:49:54.135736    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 21:50:11.028714 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:50:11.028678    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:50:14.832616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:14.832580    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"]
Oct 30 21:50:14.838421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:14.838397    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:14.852615 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:14.852591    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-5d3601-serving-cert\""
Oct 30 21:50:14.858355 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:14.858320    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"]
Oct 30 21:50:15.031135 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.031095    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-metrics-certs\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.032126 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.032096    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.032823 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.032802    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljlws\" (UniqueName: \"kubernetes.io/projected/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-kube-api-access-ljlws\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.033607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.033586    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.141184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.141059    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.141380 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.141194    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljlws\" (UniqueName: \"kubernetes.io/projected/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-kube-api-access-ljlws\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.141380 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.141239    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.141498 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.141424    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-metrics-certs\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.141942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.141914    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.143690 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.143666    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.143819 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.143755    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-metrics-certs\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.159184 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.159146    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljlws\" (UniqueName: \"kubernetes.io/projected/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-kube-api-access-ljlws\") pod \"maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.454731 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.454687    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:15.762883 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.762852    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"]
Oct 30 21:50:15.765223 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:50:15.765180    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d2b79f_b40a_411d_bf1c_9d5a101b8e7a.slice/crio-f79878b397c8757853704acb362bac09dd458374ea46c82711922950d4bad123 WatchSource:0}: Error finding container f79878b397c8757853704acb362bac09dd458374ea46c82711922950d4bad123: Status 404 returned error can't find the container with id f79878b397c8757853704acb362bac09dd458374ea46c82711922950d4bad123
Oct 30 21:50:15.766700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:15.766672    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:50:16.333142 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:16.333094    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd" event={"ID":"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a","Type":"ContainerStarted","Data":"599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89"}
Oct 30 21:50:16.333577 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:16.333198    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:16.333577 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:16.333227    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd" event={"ID":"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a","Type":"ContainerStarted","Data":"f79878b397c8757853704acb362bac09dd458374ea46c82711922950d4bad123"}
Oct 30 21:50:16.372661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:16.372197    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd" podStartSLOduration=2.023567917 podStartE2EDuration="2.372177406s" totalImagesPullingTime="348.609489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:16.370038106 +0000 UTC m=+2416.499289564" watchObservedRunningTime="2026-10-30 21:50:16.372177406 +0000 UTC m=+2416.501428863"
Oct 30 21:50:22.345941 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:50:22.345910    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 21:50:41.029584 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:50:41.029493    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:51:01.140121 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.140086    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"]
Oct 30 21:51:01.144497 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.144467    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.161141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.161109    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ed0866-openshift-default-dockercfg-sl7dd\""
Oct 30 21:51:01.167485 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.167373    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"]
Oct 30 21:51:01.278892 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.278854    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt9m4\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-kube-api-access-tt9m4\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.279312 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.279268    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.279451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.279332    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-credential-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.279451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.279377    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-certs\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.279451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.279427    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-data\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.279639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.279526    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4631b858-9891-4eda-8a4a-8142682819b3-istio-podinfo\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.279639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.279582    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4631b858-9891-4eda-8a4a-8142682819b3-istiod-ca-cert\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.279639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.279626    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-istio-token\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.279871 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.279679    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-envoy\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.381045 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.381002    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-certs\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.381775 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.381748    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-data\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.382021 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.382005    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4631b858-9891-4eda-8a4a-8142682819b3-istio-podinfo\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.382187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.382170    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4631b858-9891-4eda-8a4a-8142682819b3-istiod-ca-cert\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.382338 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.382314    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-istio-token\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.382474 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.382460    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-envoy\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.382655 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.382639    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt9m4\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-kube-api-access-tt9m4\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.382808 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.382792    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.382925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.382910    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-credential-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.383252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.383091    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4631b858-9891-4eda-8a4a-8142682819b3-istiod-ca-cert\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.383709 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.381665    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-certs\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.384115 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.384091    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.384288 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.382474    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-data\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.389470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.389446    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-credential-socket\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.404009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.403972    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt9m4\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-kube-api-access-tt9m4\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.407528 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.407477    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-istio-token\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.413274 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.413246    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4631b858-9891-4eda-8a4a-8142682819b3-istio-podinfo\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.413395 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.413280    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-envoy\") pod \"e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") " pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.461868 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.461831    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:01.801473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.801448    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"]
Oct 30 21:51:01.803450 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:51:01.803423    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4631b858_9891_4eda_8a4a_8142682819b3.slice/crio-9f350678a66617d385f58dc14dba2707fa7e9ebb6ec8837ca1aebc9a5c0118e1 WatchSource:0}: Error finding container 9f350678a66617d385f58dc14dba2707fa7e9ebb6ec8837ca1aebc9a5c0118e1: Status 404 returned error can't find the container with id 9f350678a66617d385f58dc14dba2707fa7e9ebb6ec8837ca1aebc9a5c0118e1
Oct 30 21:51:01.806782 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.806740    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:01.806877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.806818    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:01.806877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.806850    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:51:01.826822 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:01.826779    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" event={"ID":"4631b858-9891-4eda-8a4a-8142682819b3","Type":"ContainerStarted","Data":"9f350678a66617d385f58dc14dba2707fa7e9ebb6ec8837ca1aebc9a5c0118e1"}
Oct 30 21:51:02.836536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:02.836497    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" event={"ID":"4631b858-9891-4eda-8a4a-8142682819b3","Type":"ContainerStarted","Data":"b4d47d01eb1e6b446d45f641c0d1f9917acc54f8e36e8638e45fc934add0b906"}
Oct 30 21:51:02.899112 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:02.899051    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" podStartSLOduration=1.899038382 podStartE2EDuration="1.899038382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:02.887121808 +0000 UTC m=+2463.016373265" watchObservedRunningTime="2026-10-30 21:51:02.899038382 +0000 UTC m=+2463.028289839"
Oct 30 21:51:03.469178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.467563    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:03.550207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.549956    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:03.797708 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.797621    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"]
Oct 30 21:51:03.802289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.802261    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:03.816594 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.816571    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"agc-cred-c26bbe9ee934\""
Oct 30 21:51:03.816741 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.816603    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"kube-root-ca.crt\""
Oct 30 21:51:03.816807 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.816795    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"routing-overlay\""
Oct 30 21:51:03.816924 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.816908    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-plugins-e2e-extmod-b-db1288\""
Oct 30 21:51:03.816981 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.816951    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"openshift-service-ca.crt\""
Oct 30 21:51:03.817037 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.816988    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"agc-cred-44df4141bbb0\""
Oct 30 21:51:03.817037 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.817020    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-e2e--f0c0883f-dockercfg-8mgn7\""
Oct 30 21:51:03.822471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.822445    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"]
Oct 30 21:51:03.849955 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.849930    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:03.851379 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.851358    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:51:03.927333 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.927294    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:03.927830 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.927810    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq5fl\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-kube-api-access-bq5fl\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:03.928716 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.928695    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:03.928975 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:03.928959    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.036180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.031697    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq5fl\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-kube-api-access-bq5fl\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.036180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.031755    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.036180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.031854    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.036180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.031916    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.036180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.034476    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.036622 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.036588    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.038106 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.038079    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.052693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.052633    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq5fl\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-kube-api-access-bq5fl\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.120838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.120813    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.428777 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.428746    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"]
Oct 30 21:51:04.432605 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:51:04.432563    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c03041_7157_491a_80b4_301de2c68fde.slice/crio-8ea321a962769bd3ecb0ccfe6d78ddebf2623157f19af3e86cc875830567d91d WatchSource:0}: Error finding container 8ea321a962769bd3ecb0ccfe6d78ddebf2623157f19af3e86cc875830567d91d: Status 404 returned error can't find the container with id 8ea321a962769bd3ecb0ccfe6d78ddebf2623157f19af3e86cc875830567d91d
Oct 30 21:51:04.859109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.859001    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c" event={"ID":"74c03041-7157-491a-80b4-301de2c68fde","Type":"ContainerStarted","Data":"f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b"}
Oct 30 21:51:04.859109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.859054    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c" event={"ID":"74c03041-7157-491a-80b4-301de2c68fde","Type":"ContainerStarted","Data":"8ea321a962769bd3ecb0ccfe6d78ddebf2623157f19af3e86cc875830567d91d"}
Oct 30 21:51:04.859109 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.859095    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:04.897207 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:04.897131    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c" podStartSLOduration=1.897115167 podStartE2EDuration="1.897115167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:04.895860929 +0000 UTC m=+2465.025112387" watchObservedRunningTime="2026-10-30 21:51:04.897115167 +0000 UTC m=+2465.026366626"
Oct 30 21:51:11.030354 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:51:11.030313    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:51:13.616897 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:13.616855    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"]
Oct 30 21:51:15.871141 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:15.871111    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:51:18.617421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.617364    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr" podUID="99cb2c67-612a-455e-ab73-1469dc5b4097" containerName="payload-processing" containerID="cri-o://677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8" gracePeriod=25
Oct 30 21:51:18.895223 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.895193    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:51:18.989627 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.989597    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-kube-api-access-qnpxg\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-kube-api-access-qnpxg\") pod \"99cb2c67-612a-455e-ab73-1469dc5b4097\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") "
Oct 30 21:51:18.989838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.989755    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-plugins-config-volume\") pod \"99cb2c67-612a-455e-ab73-1469dc5b4097\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") "
Oct 30 21:51:18.989912 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.989845    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-provider-credentials\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-provider-credentials\") pod \"99cb2c67-612a-455e-ab73-1469dc5b4097\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") "
Oct 30 21:51:18.989975 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.989931    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-routing-overlay-volume\") pod \"99cb2c67-612a-455e-ab73-1469dc5b4097\" (UID: \"99cb2c67-612a-455e-ab73-1469dc5b4097\") "
Oct 30 21:51:18.990226 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.990195    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-plugins-config-volume" pod "99cb2c67-612a-455e-ab73-1469dc5b4097" (UID: "99cb2c67-612a-455e-ab73-1469dc5b4097"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:18.990490 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.990458    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-routing-overlay-volume" pod "99cb2c67-612a-455e-ab73-1469dc5b4097" (UID: "99cb2c67-612a-455e-ab73-1469dc5b4097"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:18.990765 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.990741    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:51:18.990855 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.990774    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/99cb2c67-612a-455e-ab73-1469dc5b4097-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:51:18.992576 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.992543    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-kube-api-access-qnpxg" pod "99cb2c67-612a-455e-ab73-1469dc5b4097" (UID: "99cb2c67-612a-455e-ab73-1469dc5b4097"). InnerVolumeSpecName "kube-api-access-qnpxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:18.992838 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.992811    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-provider-credentials" pod "99cb2c67-612a-455e-ab73-1469dc5b4097" (UID: "99cb2c67-612a-455e-ab73-1469dc5b4097"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:18.996540 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.996512    5886 generic.go:356] "Generic (PLEG): container finished" podID="99cb2c67-612a-455e-ab73-1469dc5b4097" containerID="677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8" exitCode=0
Oct 30 21:51:18.996666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.996557    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"
Oct 30 21:51:18.996666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.996593    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr" event={"ID":"99cb2c67-612a-455e-ab73-1469dc5b4097","Type":"ContainerDied","Data":"677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8"}
Oct 30 21:51:18.996666 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.996648    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr" event={"ID":"99cb2c67-612a-455e-ab73-1469dc5b4097","Type":"ContainerDied","Data":"9e82f25db614bf3b654dc56526bd36fd539df6379f38360b6b49a97436885efc"}
Oct 30 21:51:18.996836 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:18.996678    5886 scope.go:122] "RemoveContainer" containerID="677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8"
Oct 30 21:51:19.021187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:19.021144    5886 scope.go:122] "RemoveContainer" containerID="677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8"
Oct 30 21:51:19.021496 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:51:19.021477    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8\": container with ID starting with 677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8 not found: ID does not exist" containerID="677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8"
Oct 30 21:51:19.021551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:19.021511    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8"} err="failed to get container status \"677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8\": rpc error: code = NotFound desc = could not find container \"677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8\": container with ID starting with 677c46cf6ad81eb33c41335f1b61bf604b4ba1a80b99501e25263e12bcda55c8 not found: ID does not exist"
Oct 30 21:51:19.055002 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:19.054976    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"]
Oct 30 21:51:19.060840 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:19.060813    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdb168/payload-processing-external-model-e2e-extmod-b-fdb168-56f5zksqr"]
Oct 30 21:51:19.098183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:19.092452    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:51:19.098183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:19.092493    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnpxg\" (UniqueName: \"kubernetes.io/projected/99cb2c67-612a-455e-ab73-1469dc5b4097-kube-api-access-qnpxg\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:51:19.989308 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:19.989272    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99cb2c67-612a-455e-ab73-1469dc5b4097" path="/var/lib/kubelet/pods/99cb2c67-612a-455e-ab73-1469dc5b4097/volumes"
Oct 30 21:51:41.032831 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:51:41.032792    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:51:50.586592 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.586555    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"]
Oct 30 21:51:50.587254 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.587238    5886 cpu_manager.go:414] "containerMap: removing container" podUID="99cb2c67-612a-455e-ab73-1469dc5b4097" containerName="payload-processing"
Oct 30 21:51:50.587327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.587255    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cb2c67-612a-455e-ab73-1469dc5b4097" containerName="payload-processing"
Oct 30 21:51:50.587375 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.587365    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cb2c67-612a-455e-ab73-1469dc5b4097" containerName="payload-processing"
Oct 30 21:51:50.592538 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.592516    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.609213 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.609182    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"]
Oct 30 21:51:50.615939 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.615912    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"agc-cred-798b6dd47971\""
Oct 30 21:51:50.665017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.664978    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.665195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.665040    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.665195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.665097    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxg6\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-kube-api-access-2kxg6\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.665320 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.665298    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.766434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.766396    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.766630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.766455    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.766630 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.766485    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxg6\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-kube-api-access-2kxg6\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.766737 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.766678    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.767210 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.767181    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.767576 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.767553    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.769566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.769536    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.785644 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.785611    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxg6\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-kube-api-access-2kxg6\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:50.909470 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:50.909435    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:51.260986 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:51.260902    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"]
Oct 30 21:51:51.264793 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:51:51.264768    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bf3916_5580_4a42_846e_d2e805638d9f.slice/crio-1e3c7d651570425e27f4a957d4b764a31936a633e53c1ba3da2e724a8357ab1d WatchSource:0}: Error finding container 1e3c7d651570425e27f4a957d4b764a31936a633e53c1ba3da2e724a8357ab1d: Status 404 returned error can't find the container with id 1e3c7d651570425e27f4a957d4b764a31936a633e53c1ba3da2e724a8357ab1d
Oct 30 21:51:52.266554 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:52.266519    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42" event={"ID":"40bf3916-5580-4a42-846e-d2e805638d9f","Type":"ContainerStarted","Data":"55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986"}
Oct 30 21:51:52.266977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:52.266567    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42" event={"ID":"40bf3916-5580-4a42-846e-d2e805638d9f","Type":"ContainerStarted","Data":"1e3c7d651570425e27f4a957d4b764a31936a633e53c1ba3da2e724a8357ab1d"}
Oct 30 21:51:52.266977 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:52.266631    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:51:52.311125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:51:52.311075    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42" podStartSLOduration=2.311058219 podStartE2EDuration="2.311058219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:52.308407935 +0000 UTC m=+2512.437659404" watchObservedRunningTime="2026-10-30 21:51:52.311058219 +0000 UTC m=+2512.440309898"
Oct 30 21:52:03.278108 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:03.278075    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:52:03.353802 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:03.353762    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"]
Oct 30 21:52:08.355140 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.355095    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c" podUID="74c03041-7157-491a-80b4-301de2c68fde" containerName="payload-processing" containerID="cri-o://f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b" gracePeriod=25
Oct 30 21:52:08.634548 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.634522    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:52:08.753925 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.753882    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-plugins-config-volume\") pod \"74c03041-7157-491a-80b4-301de2c68fde\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") "
Oct 30 21:52:08.754144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.753958    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-provider-credentials\") pod \"74c03041-7157-491a-80b4-301de2c68fde\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") "
Oct 30 21:52:08.754144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.754040    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-kube-api-access-bq5fl\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-kube-api-access-bq5fl\") pod \"74c03041-7157-491a-80b4-301de2c68fde\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") "
Oct 30 21:52:08.754144 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.754082    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-routing-overlay-volume\") pod \"74c03041-7157-491a-80b4-301de2c68fde\" (UID: \"74c03041-7157-491a-80b4-301de2c68fde\") "
Oct 30 21:52:08.754364 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.754305    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-plugins-config-volume" pod "74c03041-7157-491a-80b4-301de2c68fde" (UID: "74c03041-7157-491a-80b4-301de2c68fde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:08.754509 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.754485    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-routing-overlay-volume" pod "74c03041-7157-491a-80b4-301de2c68fde" (UID: "74c03041-7157-491a-80b4-301de2c68fde"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:08.754791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.754763    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:52:08.754791 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.754782    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c03041-7157-491a-80b4-301de2c68fde-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:52:08.756229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.756205    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-kube-api-access-bq5fl" pod "74c03041-7157-491a-80b4-301de2c68fde" (UID: "74c03041-7157-491a-80b4-301de2c68fde"). InnerVolumeSpecName "kube-api-access-bq5fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:08.756306 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.756268    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-provider-credentials" pod "74c03041-7157-491a-80b4-301de2c68fde" (UID: "74c03041-7157-491a-80b4-301de2c68fde"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:08.856038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.856000    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:52:08.856038 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:08.856027    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq5fl\" (UniqueName: \"kubernetes.io/projected/74c03041-7157-491a-80b4-301de2c68fde-kube-api-access-bq5fl\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:52:09.401129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.401091    5886 generic.go:356] "Generic (PLEG): container finished" podID="74c03041-7157-491a-80b4-301de2c68fde" containerID="f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b" exitCode=0
Oct 30 21:52:09.401129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.401132    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c" event={"ID":"74c03041-7157-491a-80b4-301de2c68fde","Type":"ContainerDied","Data":"f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b"}
Oct 30 21:52:09.401653 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.401142    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"
Oct 30 21:52:09.401653 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.401193    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c" event={"ID":"74c03041-7157-491a-80b4-301de2c68fde","Type":"ContainerDied","Data":"8ea321a962769bd3ecb0ccfe6d78ddebf2623157f19af3e86cc875830567d91d"}
Oct 30 21:52:09.401653 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.401217    5886 scope.go:122] "RemoveContainer" containerID="f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b"
Oct 30 21:52:09.412584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.412567    5886 scope.go:122] "RemoveContainer" containerID="f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b"
Oct 30 21:52:09.412882 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:52:09.412862    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b\": container with ID starting with f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b not found: ID does not exist" containerID="f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b"
Oct 30 21:52:09.412960 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.412890    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b"} err="failed to get container status \"f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b\": rpc error: code = NotFound desc = could not find container \"f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b\": container with ID starting with f625f663ae27be3953155103c98eeb6032e7c4da9c01b1c6b61f940b1768ab5b not found: ID does not exist"
Oct 30 21:52:09.453009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.452979    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"]
Oct 30 21:52:09.477179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:09.473720    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-69889jb9c"]
Oct 30 21:52:10.002579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:10.002536    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c03041-7157-491a-80b4-301de2c68fde" path="/var/lib/kubelet/pods/74c03041-7157-491a-80b4-301de2c68fde/volumes"
Oct 30 21:52:11.030503 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:52:11.030419    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:52:18.495853 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:18.495761    5886 generic.go:356] "Generic (PLEG): container finished" podID="4631b858-9891-4eda-8a4a-8142682819b3" containerID="b4d47d01eb1e6b446d45f641c0d1f9917acc54f8e36e8638e45fc934add0b906" exitCode=0
Oct 30 21:52:18.496385 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:18.495957    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" event={"ID":"4631b858-9891-4eda-8a4a-8142682819b3","Type":"ContainerDied","Data":"b4d47d01eb1e6b446d45f641c0d1f9917acc54f8e36e8638e45fc934add0b906"}
Oct 30 21:52:18.496525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:18.496503    5886 scope.go:122] "RemoveContainer" containerID="b4d47d01eb1e6b446d45f641c0d1f9917acc54f8e36e8638e45fc934add0b906"
Oct 30 21:52:18.498903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:18.498867    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:18.499009 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:18.498957    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:18.499049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:18.499005    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:52:18.850833 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:18.850751    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:52:19.508318 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:19.508275    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" event={"ID":"4631b858-9891-4eda-8a4a-8142682819b3","Type":"ContainerStarted","Data":"1a76c3916e102a532acf30cf81aa9f9992befde7fd6bff3810f474e7f223cfc4"}
Oct 30 21:52:20.467181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:20.466223    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:52:20.471188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:20.470252    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:52:20.471188 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:20.470326    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 21:52:21.465567 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:21.465527    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:52:21.466889 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:21.466851    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 21:52:22.466603 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:22.466567    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:52:22.482355 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:22.478606    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 21:52:23.466181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:23.465503    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:52:23.466181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:23.465578    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 21:52:24.475283 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:24.475244    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:52:24.476342 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:24.475842    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 21:52:25.467956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:25.467919    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:52:25.468442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:25.468415    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:52:25.469202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:25.469183    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 21:52:41.029313 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:52:41.029275    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:52:58.986324 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:52:58.986293    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"]
Oct 30 21:53:00.369973 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:00.369928    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"]
Oct 30 21:53:00.370484 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:00.370183    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy" containerID="cri-o://7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931" gracePeriod=30
Oct 30 21:53:03.986681 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:03.986621    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42" podUID="40bf3916-5580-4a42-846e-d2e805638d9f" containerName="payload-processing" containerID="cri-o://55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986" gracePeriod=25
Oct 30 21:53:04.348252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.348226    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:53:04.478978 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.478940    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-provider-credentials\") pod \"40bf3916-5580-4a42-846e-d2e805638d9f\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") "
Oct 30 21:53:04.480804 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.479342    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-kube-api-access-2kxg6\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-kube-api-access-2kxg6\") pod \"40bf3916-5580-4a42-846e-d2e805638d9f\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") "
Oct 30 21:53:04.481518 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.481497    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-routing-overlay-volume\") pod \"40bf3916-5580-4a42-846e-d2e805638d9f\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") "
Oct 30 21:53:04.482168 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.481997    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-routing-overlay-volume" pod "40bf3916-5580-4a42-846e-d2e805638d9f" (UID: "40bf3916-5580-4a42-846e-d2e805638d9f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:04.482357 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.482341    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-plugins-config-volume\") pod \"40bf3916-5580-4a42-846e-d2e805638d9f\" (UID: \"40bf3916-5580-4a42-846e-d2e805638d9f\") "
Oct 30 21:53:04.482970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.482859    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-plugins-config-volume" pod "40bf3916-5580-4a42-846e-d2e805638d9f" (UID: "40bf3916-5580-4a42-846e-d2e805638d9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:04.483855 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.483834    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:04.484034 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.484019    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40bf3916-5580-4a42-846e-d2e805638d9f-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:04.515059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.515022    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-kube-api-access-2kxg6" pod "40bf3916-5580-4a42-846e-d2e805638d9f" (UID: "40bf3916-5580-4a42-846e-d2e805638d9f"). InnerVolumeSpecName "kube-api-access-2kxg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:04.517686 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.517651    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-provider-credentials" pod "40bf3916-5580-4a42-846e-d2e805638d9f" (UID: "40bf3916-5580-4a42-846e-d2e805638d9f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:04.616546 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.616456    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kxg6\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-kube-api-access-2kxg6\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:04.616546 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:04.616495    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40bf3916-5580-4a42-846e-d2e805638d9f-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.029695 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.029649    5886 generic.go:356] "Generic (PLEG): container finished" podID="40bf3916-5580-4a42-846e-d2e805638d9f" containerID="55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986" exitCode=0
Oct 30 21:53:05.030203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.029762    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"
Oct 30 21:53:05.030203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.029764    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42" event={"ID":"40bf3916-5580-4a42-846e-d2e805638d9f","Type":"ContainerDied","Data":"55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986"}
Oct 30 21:53:05.030203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.029836    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42" event={"ID":"40bf3916-5580-4a42-846e-d2e805638d9f","Type":"ContainerDied","Data":"1e3c7d651570425e27f4a957d4b764a31936a633e53c1ba3da2e724a8357ab1d"}
Oct 30 21:53:05.030203 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.029871    5886 scope.go:122] "RemoveContainer" containerID="55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986"
Oct 30 21:53:05.042441 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.042419    5886 scope.go:122] "RemoveContainer" containerID="55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986"
Oct 30 21:53:05.042785 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:53:05.042766    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986\": container with ID starting with 55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986 not found: ID does not exist" containerID="55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986"
Oct 30 21:53:05.042843 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.042793    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986"} err="failed to get container status \"55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986\": rpc error: code = NotFound desc = could not find container \"55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986\": container with ID starting with 55ce8e9073344c824abd5be6542dc5849c2168a575a367d556d3e1f4d5dd8986 not found: ID does not exist"
Oct 30 21:53:05.086358 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.086317    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"]
Oct 30 21:53:05.095172 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.095133    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-b9d4mqv42"]
Oct 30 21:53:05.717317 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.717289    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:53:05.732260 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732231    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-certs\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732275    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-data\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732312    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-kube-api-access-vk8tq\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732383    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-credential-socket\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732407 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732405    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-envoy\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732424    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd02068-a0b4-44b0-905c-bd4e0e650977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd02068-a0b4-44b0-905c-bd4e0e650977-istiod-ca-cert\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732483    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-podinfo\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732524    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-certs" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:05.732610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732546    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-data" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:05.732610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732536    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-token\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-token\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732653    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-socket\") pod \"ccd02068-a0b4-44b0-905c-bd4e0e650977\" (UID: \"ccd02068-a0b4-44b0-905c-bd4e0e650977\") "
Oct 30 21:53:05.732851 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732690    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-credential-socket" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:05.732951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.732848    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd02068-a0b4-44b0-905c-bd4e0e650977-istiod-ca-cert" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:05.733307 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.733284    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-socket" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:05.733447 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.733289    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.733533 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.733523    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.733614 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.733604    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.733701 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.733691    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd02068-a0b4-44b0-905c-bd4e0e650977-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.734864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.734812    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-podinfo" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:53:05.734864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.734851    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-token" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:05.735874 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.735837    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-envoy" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:05.736108 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.736083    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-kube-api-access-vk8tq" pod "ccd02068-a0b4-44b0-905c-bd4e0e650977" (UID: "ccd02068-a0b4-44b0-905c-bd4e0e650977"). InnerVolumeSpecName "kube-api-access-vk8tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:05.835679 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.835647    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.835679 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.835679    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk8tq\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-kube-api-access-vk8tq\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.835891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.835695    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.835891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.835712    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.835891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.835725    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd02068-a0b4-44b0-905c-bd4e0e650977-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:53:05.993370 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:05.991144    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bf3916-5580-4a42-846e-d2e805638d9f" path="/var/lib/kubelet/pods/40bf3916-5580-4a42-846e-d2e805638d9f/volumes"
Oct 30 21:53:06.039434 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.039401    5886 generic.go:356] "Generic (PLEG): container finished" podID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerID="7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931" exitCode=0
Oct 30 21:53:06.039903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.039468    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"
Oct 30 21:53:06.039903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.039478    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" event={"ID":"ccd02068-a0b4-44b0-905c-bd4e0e650977","Type":"ContainerDied","Data":"7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931"}
Oct 30 21:53:06.039903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.039518    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr" event={"ID":"ccd02068-a0b4-44b0-905c-bd4e0e650977","Type":"ContainerDied","Data":"625fcb64f2470387d8e67b77ab5da8ae0f01f2631cd674cd1eb33af6174210ab"}
Oct 30 21:53:06.039903 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.039538    5886 scope.go:122] "RemoveContainer" containerID="7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931"
Oct 30 21:53:06.049632 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.049610    5886 scope.go:122] "RemoveContainer" containerID="7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931"
Oct 30 21:53:06.049949 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:53:06.049921    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931\": container with ID starting with 7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931 not found: ID does not exist" containerID="7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931"
Oct 30 21:53:06.050049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.049960    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931"} err="failed to get container status \"7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931\": rpc error: code = NotFound desc = could not find container \"7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931\": container with ID starting with 7a4a0dfdc764a8c0b2f5a4bf49be607e9f85230432cb2d68b9179d8e06d83931 not found: ID does not exist"
Oct 30 21:53:06.090605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.090576    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"]
Oct 30 21:53:06.101565 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:06.101533    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b31518-openshift-default-7b45c7c84c-ppntr"]
Oct 30 21:53:07.991646 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:07.991609    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" path="/var/lib/kubelet/pods/ccd02068-a0b4-44b0-905c-bd4e0e650977/volumes"
Oct 30 21:53:11.065692 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:53:11.065653    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:53:11.351906 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.351821    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"]
Oct 30 21:53:11.352496 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352482    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy"
Oct 30 21:53:11.352549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352497    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy"
Oct 30 21:53:11.352549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352526    5886 cpu_manager.go:414] "containerMap: removing container" podUID="74c03041-7157-491a-80b4-301de2c68fde" containerName="payload-processing"
Oct 30 21:53:11.352549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352531    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c03041-7157-491a-80b4-301de2c68fde" containerName="payload-processing"
Oct 30 21:53:11.352549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352538    5886 cpu_manager.go:414] "containerMap: removing container" podUID="40bf3916-5580-4a42-846e-d2e805638d9f" containerName="payload-processing"
Oct 30 21:53:11.352549 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352544    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bf3916-5580-4a42-846e-d2e805638d9f" containerName="payload-processing"
Oct 30 21:53:11.352704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352643    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c03041-7157-491a-80b4-301de2c68fde" containerName="payload-processing"
Oct 30 21:53:11.352704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352652    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bf3916-5580-4a42-846e-d2e805638d9f" containerName="payload-processing"
Oct 30 21:53:11.352704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.352664    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd02068-a0b4-44b0-905c-bd4e0e650977" containerName="istio-proxy"
Oct 30 21:53:11.357704 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.357685    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.383934 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.383906    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"kube-root-ca.crt\""
Oct 30 21:53:11.384193 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.384171    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"openshift-service-ca.crt\""
Oct 30 21:53:11.384342 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.384182    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"agc-cred-c26bbe9ee934\""
Oct 30 21:53:11.384415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.384204    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-e2e--f0c0883f-dockercfg-8mgn7\""
Oct 30 21:53:11.384415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.384244    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"routing-overlay\""
Oct 30 21:53:11.384415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.384278    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-plugins-e2e-extmod-b-db1288\""
Oct 30 21:53:11.390084 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.390059    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"]
Oct 30 21:53:11.415233 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.415207    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t69wp\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-kube-api-access-t69wp\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.415376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.415252    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.415376 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.415343    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.415492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.415414    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.517216 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.517180    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t69wp\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-kube-api-access-t69wp\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.517216 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.517222    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.517424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.517261    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.517424 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.517326    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.518030 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.518003    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.518206 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.518110    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.519804 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.519774    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.536580 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.536553    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t69wp\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-kube-api-access-t69wp\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.672605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.672569    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:11.996077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:11.995872    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"]
Oct 30 21:53:11.999545 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:53:11.999514    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74846053_7513_4018_8a91_67cad9c87020.slice/crio-946f3d1621b9324344947bbc730f07d4c8cf16abef4f91b5bb6fd449eb666986 WatchSource:0}: Error finding container 946f3d1621b9324344947bbc730f07d4c8cf16abef4f91b5bb6fd449eb666986: Status 404 returned error can't find the container with id 946f3d1621b9324344947bbc730f07d4c8cf16abef4f91b5bb6fd449eb666986
Oct 30 21:53:12.127748 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:12.127715    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf" event={"ID":"74846053-7513-4018-8a91-67cad9c87020","Type":"ContainerStarted","Data":"946f3d1621b9324344947bbc730f07d4c8cf16abef4f91b5bb6fd449eb666986"}
Oct 30 21:53:13.137226 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.137180    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf" event={"ID":"74846053-7513-4018-8a91-67cad9c87020","Type":"ContainerStarted","Data":"3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a"}
Oct 30 21:53:13.137639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.137251    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:13.179771 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.179721    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf" podStartSLOduration=2.179707263 podStartE2EDuration="2.179707263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:13.176048343 +0000 UTC m=+2593.305299801" watchObservedRunningTime="2026-10-30 21:53:13.179707263 +0000 UTC m=+2593.308958721"
Oct 30 21:53:13.624632 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.624590    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"]
Oct 30 21:53:13.631954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.631932    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.646386 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.646362    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-805edc-openshift-default-dockercfg-vvclz\""
Oct 30 21:53:13.666698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.666667    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"]
Oct 30 21:53:13.744866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.744827    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.745041 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.744969    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-data\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.745041 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.745033    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbkk\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-kube-api-access-bzbkk\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.745117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.745054    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-podinfo\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.745202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.745149    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istiod-ca-cert\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.745263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.745252    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-envoy\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.745296 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.745274    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-certs\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.745326 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.745310    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-token\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.745390 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.745372    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-credential-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.846642 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.846607    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-data\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.846852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.846668    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbkk\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-kube-api-access-bzbkk\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.846852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.846690    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-podinfo\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.846852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.846711    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istiod-ca-cert\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.846852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.846753    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-envoy\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.846852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.846789    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-certs\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.847148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.846862    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-token\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.847148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.846897    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-credential-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.847148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.847063    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.847384 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.847064    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-data\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.847451 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.847432    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-credential-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.847512 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.847455    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istiod-ca-cert\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.847660 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.847637    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-certs\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.847735 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.847676    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-socket\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.849099 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.849069    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-envoy\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.849252 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.849183    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-podinfo\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.869054 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.869026    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-token\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.869719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.869684    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbkk\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-kube-api-access-bzbkk\") pod \"e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") " pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:13.946967 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:13.946928    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:14.300390 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:53:14.300356    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f1c731_26c5_4ec4_abd8_458909a6ff6e.slice/crio-366727466fe6aa41b437fe020cbced50653b7560bf7a3467d07c8d5c733adfc9 WatchSource:0}: Error finding container 366727466fe6aa41b437fe020cbced50653b7560bf7a3467d07c8d5c733adfc9: Status 404 returned error can't find the container with id 366727466fe6aa41b437fe020cbced50653b7560bf7a3467d07c8d5c733adfc9
Oct 30 21:53:14.306025 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:14.305981    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:14.306185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:14.306075    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:14.306185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:14.306117    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:53:14.306309 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:14.306278    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"]
Oct 30 21:53:15.152868 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:15.152833    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" event={"ID":"b9f1c731-26c5-4ec4-abd8-458909a6ff6e","Type":"ContainerStarted","Data":"2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2"}
Oct 30 21:53:15.153079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:15.152889    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" event={"ID":"b9f1c731-26c5-4ec4-abd8-458909a6ff6e","Type":"ContainerStarted","Data":"366727466fe6aa41b437fe020cbced50653b7560bf7a3467d07c8d5c733adfc9"}
Oct 30 21:53:15.220251 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:15.220198    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" podStartSLOduration=2.220184257 podStartE2EDuration="2.220184257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:15.209143646 +0000 UTC m=+2595.338395104" watchObservedRunningTime="2026-10-30 21:53:15.220184257 +0000 UTC m=+2595.349435713"
Oct 30 21:53:15.947607 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:15.947575    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:15.948714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:15.948689    5886 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 21:53:15.948860 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:15.948733    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 21:53:16.947787 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:16.947754    5886 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 21:53:16.948244 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:16.947815    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 21:53:17.947691 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:17.947653    5886 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 21:53:17.947875 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:17.947713    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 21:53:18.947473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:18.947427    5886 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 21:53:18.947643 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:18.947488    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 21:53:19.948076 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:19.948043    5886 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 21:53:19.948494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:19.948092    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 21:53:21.019597 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:21.019558    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:21.020611 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:21.020580    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:21.022231 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:21.022207    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 21:53:24.147303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:24.147273    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:53:41.029704 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:53:41.029624    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:53:56.201736 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:53:56.201704    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"]
Oct 30 21:54:01.202226 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.202162    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf" podUID="74846053-7513-4018-8a91-67cad9c87020" containerName="payload-processing" containerID="cri-o://3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a" gracePeriod=25
Oct 30 21:54:01.484202 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.484171    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:54:01.562999 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.562966    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-plugins-config-volume\") pod \"74846053-7513-4018-8a91-67cad9c87020\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") "
Oct 30 21:54:01.563196 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.563143    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-routing-overlay-volume\") pod \"74846053-7513-4018-8a91-67cad9c87020\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") "
Oct 30 21:54:01.563277 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.563219    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-provider-credentials\") pod \"74846053-7513-4018-8a91-67cad9c87020\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") "
Oct 30 21:54:01.563341 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.563279    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-kube-api-access-t69wp\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-kube-api-access-t69wp\") pod \"74846053-7513-4018-8a91-67cad9c87020\" (UID: \"74846053-7513-4018-8a91-67cad9c87020\") "
Oct 30 21:54:01.563492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.563443    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-plugins-config-volume" pod "74846053-7513-4018-8a91-67cad9c87020" (UID: "74846053-7513-4018-8a91-67cad9c87020"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:01.563756 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.563690    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-routing-overlay-volume" pod "74846053-7513-4018-8a91-67cad9c87020" (UID: "74846053-7513-4018-8a91-67cad9c87020"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:01.564015 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.563992    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:01.564133 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.564017    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74846053-7513-4018-8a91-67cad9c87020-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:01.566114 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.566089    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-provider-credentials" pod "74846053-7513-4018-8a91-67cad9c87020" (UID: "74846053-7513-4018-8a91-67cad9c87020"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:01.566331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.566116    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-kube-api-access-t69wp" pod "74846053-7513-4018-8a91-67cad9c87020" (UID: "74846053-7513-4018-8a91-67cad9c87020"). InnerVolumeSpecName "kube-api-access-t69wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:01.586526 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.586497    5886 generic.go:356] "Generic (PLEG): container finished" podID="74846053-7513-4018-8a91-67cad9c87020" containerID="3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a" exitCode=0
Oct 30 21:54:01.586663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.586543    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"
Oct 30 21:54:01.586663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.586570    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf" event={"ID":"74846053-7513-4018-8a91-67cad9c87020","Type":"ContainerDied","Data":"3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a"}
Oct 30 21:54:01.586663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.586625    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf" event={"ID":"74846053-7513-4018-8a91-67cad9c87020","Type":"ContainerDied","Data":"946f3d1621b9324344947bbc730f07d4c8cf16abef4f91b5bb6fd449eb666986"}
Oct 30 21:54:01.586663 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.586654    5886 scope.go:122] "RemoveContainer" containerID="3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a"
Oct 30 21:54:01.600745 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.600708    5886 scope.go:122] "RemoveContainer" containerID="3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a"
Oct 30 21:54:01.601327 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:54:01.601252    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a\": container with ID starting with 3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a not found: ID does not exist" containerID="3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a"
Oct 30 21:54:01.601327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.601292    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a"} err="failed to get container status \"3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a\": rpc error: code = NotFound desc = could not find container \"3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a\": container with ID starting with 3e8bcafea27cc310b1919311b2bd353af6d1649b900ffdd75d4c4043b273fb2a not found: ID does not exist"
Oct 30 21:54:01.638060 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.638033    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"]
Oct 30 21:54:01.643561 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.643539    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-55b6xjxpf"]
Oct 30 21:54:01.664987 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.664962    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:01.664987 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:01.664987    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t69wp\" (UniqueName: \"kubernetes.io/projected/74846053-7513-4018-8a91-67cad9c87020-kube-api-access-t69wp\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:02.004292 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:02.004254    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74846053-7513-4018-8a91-67cad9c87020" path="/var/lib/kubelet/pods/74846053-7513-4018-8a91-67cad9c87020/volumes"
Oct 30 21:54:11.057576 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:54:11.057525    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:54:41.034983 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:54:41.034937    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:54:43.099276 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.099243    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"]
Oct 30 21:54:43.099905 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.099891    5886 cpu_manager.go:414] "containerMap: removing container" podUID="74846053-7513-4018-8a91-67cad9c87020" containerName="payload-processing"
Oct 30 21:54:43.099951 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.099906    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74846053-7513-4018-8a91-67cad9c87020" containerName="payload-processing"
Oct 30 21:54:43.100022 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.100008    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="74846053-7513-4018-8a91-67cad9c87020" containerName="payload-processing"
Oct 30 21:54:43.104045 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.104023    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.119712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.119691    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"openshift-service-ca.crt\""
Oct 30 21:54:43.124891 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.124861    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"]
Oct 30 21:54:43.130328 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.130304    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5d3601\""
Oct 30 21:54:43.130711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.130689    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"kube-root-ca.crt\""
Oct 30 21:54:43.130985 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.130955    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"routing-overlay\""
Oct 30 21:54:43.131375 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.131351    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"agc-cred-06c59bf05a37\""
Oct 30 21:54:43.131507 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.131465    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5d3601\"/\"payload-processing-external-model-e2e--d5cdf106-dockercfg-lhhbb\""
Oct 30 21:54:43.267961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.267911    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxj7k\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-kube-api-access-sxj7k\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.268148 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.268025    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.268270 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.268254    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.268330 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.268280    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.375219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.375071    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.375219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.375133    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.375473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.375314    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxj7k\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-kube-api-access-sxj7k\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.375473 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.375374    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.377576 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.376774    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.377576 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.377374    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.389927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.389893    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.402108 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.399810    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxj7k\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-kube-api-access-sxj7k\") pod \"payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") " pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.418767 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.418723    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:43.736096 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:43.736071    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"]
Oct 30 21:54:43.738470 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:54:43.738437    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1943b543_d739_4742_955c_2a2f8fbc8f53.slice/crio-cf307996f56e0fb4d61e5cab3c48413de5f11c25529b71da9267c7b2359963ee WatchSource:0}: Error finding container cf307996f56e0fb4d61e5cab3c48413de5f11c25529b71da9267c7b2359963ee: Status 404 returned error can't find the container with id cf307996f56e0fb4d61e5cab3c48413de5f11c25529b71da9267c7b2359963ee
Oct 30 21:54:44.051016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.050932    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25" event={"ID":"1943b543-d739-4742-955c-2a2f8fbc8f53","Type":"ContainerStarted","Data":"4e0788ed756cdb6c31a6c197775829c924596a274822d9ee54b6c8ede4f38476"}
Oct 30 21:54:44.051016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.050993    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:44.051016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.051017    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25" event={"ID":"1943b543-d739-4742-955c-2a2f8fbc8f53","Type":"ContainerStarted","Data":"cf307996f56e0fb4d61e5cab3c48413de5f11c25529b71da9267c7b2359963ee"}
Oct 30 21:54:44.090204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.090117    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25" podStartSLOduration=1.090099952 podStartE2EDuration="1.090099952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:44.088523226 +0000 UTC m=+2684.217774683" watchObservedRunningTime="2026-10-30 21:54:44.090099952 +0000 UTC m=+2684.219351410"
Oct 30 21:54:44.693520 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.693482    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"]
Oct 30 21:54:44.698042 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.698019    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.714833 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.714804    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"kube-root-ca.crt\""
Oct 30 21:54:44.729181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.725969    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ed0866\""
Oct 30 21:54:44.729181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.725991    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"openshift-service-ca.crt\""
Oct 30 21:54:44.729181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.726271    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"agc-cred-90d1425711a8\""
Oct 30 21:54:44.729181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.726321    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"routing-overlay\""
Oct 30 21:54:44.729181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.726411    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"agc-cred-ef4ead2a98f4\""
Oct 30 21:54:44.729181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.726517    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-e2e--68c3d057-dockercfg-zz6jd\""
Oct 30 21:54:44.731927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.731901    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"]
Oct 30 21:54:44.794273 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.794244    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.794403 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.794282    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.794403 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.794313    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.794482 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.794446    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql729\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-kube-api-access-ql729\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.896289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.896249    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.896289 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.896289    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.896535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.896310    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.896535 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.896394    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql729\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-kube-api-access-ql729\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.897094 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.897065    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.897288 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.897094    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.898935 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.898914    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:44.915846 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:44.915823    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql729\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-kube-api-access-ql729\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:45.012798 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:45.012700    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:45.327049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:45.327015    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"]
Oct 30 21:54:45.328349 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:54:45.328308    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6232338a_a078_4617_b66f_00423bbf1321.slice/crio-7ce5ac8651d8bfee22265703963fbcb3b38864d69f692f1336e98629be2f2fa0 WatchSource:0}: Error finding container 7ce5ac8651d8bfee22265703963fbcb3b38864d69f692f1336e98629be2f2fa0: Status 404 returned error can't find the container with id 7ce5ac8651d8bfee22265703963fbcb3b38864d69f692f1336e98629be2f2fa0
Oct 30 21:54:46.070444 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:46.070402    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc" event={"ID":"6232338a-a078-4617-b66f-00423bbf1321","Type":"ContainerStarted","Data":"55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0"}
Oct 30 21:54:46.070971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:46.070472    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc" event={"ID":"6232338a-a078-4617-b66f-00423bbf1321","Type":"ContainerStarted","Data":"7ce5ac8651d8bfee22265703963fbcb3b38864d69f692f1336e98629be2f2fa0"}
Oct 30 21:54:46.070971 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:46.070685    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:46.109590 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:46.109532    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc" podStartSLOduration=2.109516278 podStartE2EDuration="2.109516278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:46.106767397 +0000 UTC m=+2686.236018853" watchObservedRunningTime="2026-10-30 21:54:46.109516278 +0000 UTC m=+2686.238767784"
Oct 30 21:54:48.291227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:48.291183    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"]
Oct 30 21:54:48.840221 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:48.840187    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"]
Oct 30 21:54:48.843458 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:48.843432    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:53.294296 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.294247    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc" podUID="6232338a-a078-4617-b66f-00423bbf1321" containerName="payload-processing" containerID="cri-o://55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0" gracePeriod=25
Oct 30 21:54:53.622005 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.621977    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:53.694031 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.693993    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-routing-overlay-volume\") pod \"6232338a-a078-4617-b66f-00423bbf1321\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") "
Oct 30 21:54:53.694263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.694058    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-kube-api-access-ql729\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-kube-api-access-ql729\") pod \"6232338a-a078-4617-b66f-00423bbf1321\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") "
Oct 30 21:54:53.694263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.694205    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-plugins-config-volume\") pod \"6232338a-a078-4617-b66f-00423bbf1321\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") "
Oct 30 21:54:53.694405 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.694325    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-provider-credentials\") pod \"6232338a-a078-4617-b66f-00423bbf1321\" (UID: \"6232338a-a078-4617-b66f-00423bbf1321\") "
Oct 30 21:54:53.694530 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.694432    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-routing-overlay-volume" pod "6232338a-a078-4617-b66f-00423bbf1321" (UID: "6232338a-a078-4617-b66f-00423bbf1321"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:53.694961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.694922    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-plugins-config-volume" pod "6232338a-a078-4617-b66f-00423bbf1321" (UID: "6232338a-a078-4617-b66f-00423bbf1321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:53.694961 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.694945    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:53.697227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.697197    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-provider-credentials" pod "6232338a-a078-4617-b66f-00423bbf1321" (UID: "6232338a-a078-4617-b66f-00423bbf1321"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:53.697338 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.697239    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-kube-api-access-ql729" pod "6232338a-a078-4617-b66f-00423bbf1321" (UID: "6232338a-a078-4617-b66f-00423bbf1321"). InnerVolumeSpecName "kube-api-access-ql729". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:53.796853 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.796813    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:53.796853 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.796846    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql729\" (UniqueName: \"kubernetes.io/projected/6232338a-a078-4617-b66f-00423bbf1321-kube-api-access-ql729\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:53.796853 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.796861    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6232338a-a078-4617-b66f-00423bbf1321-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:53.840697 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:53.840590    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25" podUID="1943b543-d739-4742-955c-2a2f8fbc8f53" containerName="payload-processing" containerID="cri-o://4e0788ed756cdb6c31a6c197775829c924596a274822d9ee54b6c8ede4f38476" gracePeriod=25
Oct 30 21:54:54.168179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.165534    5886 generic.go:356] "Generic (PLEG): container finished" podID="6232338a-a078-4617-b66f-00423bbf1321" containerID="55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0" exitCode=0
Oct 30 21:54:54.168179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.165805    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc" event={"ID":"6232338a-a078-4617-b66f-00423bbf1321","Type":"ContainerDied","Data":"55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0"}
Oct 30 21:54:54.168179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.165856    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc" event={"ID":"6232338a-a078-4617-b66f-00423bbf1321","Type":"ContainerDied","Data":"7ce5ac8651d8bfee22265703963fbcb3b38864d69f692f1336e98629be2f2fa0"}
Oct 30 21:54:54.168179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.165885    5886 scope.go:122] "RemoveContainer" containerID="55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0"
Oct 30 21:54:54.168179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.166036    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"
Oct 30 21:54:54.170853 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.170792    5886 generic.go:356] "Generic (PLEG): container finished" podID="1943b543-d739-4742-955c-2a2f8fbc8f53" containerID="4e0788ed756cdb6c31a6c197775829c924596a274822d9ee54b6c8ede4f38476" exitCode=0
Oct 30 21:54:54.170979 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.170867    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25" event={"ID":"1943b543-d739-4742-955c-2a2f8fbc8f53","Type":"ContainerDied","Data":"4e0788ed756cdb6c31a6c197775829c924596a274822d9ee54b6c8ede4f38476"}
Oct 30 21:54:54.184124 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.183888    5886 scope.go:122] "RemoveContainer" containerID="55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0"
Oct 30 21:54:54.184509 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:54:54.184423    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0\": container with ID starting with 55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0 not found: ID does not exist" containerID="55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0"
Oct 30 21:54:54.184509 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.184458    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0"} err="failed to get container status \"55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0\": rpc error: code = NotFound desc = could not find container \"55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0\": container with ID starting with 55f7ee5fd9162be0ed681c0770d47a249dc8acb4cbe1989d60dfe3faa5bee2e0 not found: ID does not exist"
Oct 30 21:54:54.232658 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.232625    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"]
Oct 30 21:54:54.238077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.238051    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-7b8dhzwgc"]
Oct 30 21:54:54.251347 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.251322    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:54.311295 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.311263    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-routing-overlay-volume\") pod \"1943b543-d739-4742-955c-2a2f8fbc8f53\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") "
Oct 30 21:54:54.311753 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.311466    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-provider-credentials\") pod \"1943b543-d739-4742-955c-2a2f8fbc8f53\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") "
Oct 30 21:54:54.311753 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.311548    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-kube-api-access-sxj7k\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-kube-api-access-sxj7k\") pod \"1943b543-d739-4742-955c-2a2f8fbc8f53\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") "
Oct 30 21:54:54.311753 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.311566    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-routing-overlay-volume" pod "1943b543-d739-4742-955c-2a2f8fbc8f53" (UID: "1943b543-d739-4742-955c-2a2f8fbc8f53"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:54.311753 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.311647    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-plugins-config-volume\") pod \"1943b543-d739-4742-955c-2a2f8fbc8f53\" (UID: \"1943b543-d739-4742-955c-2a2f8fbc8f53\") "
Oct 30 21:54:54.312015 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.311991    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-plugins-config-volume" pod "1943b543-d739-4742-955c-2a2f8fbc8f53" (UID: "1943b543-d739-4742-955c-2a2f8fbc8f53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:54.312263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.312243    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:54.312263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.312269    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1943b543-d739-4742-955c-2a2f8fbc8f53-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:54.314579 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.314552    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-kube-api-access-sxj7k" pod "1943b543-d739-4742-955c-2a2f8fbc8f53" (UID: "1943b543-d739-4742-955c-2a2f8fbc8f53"). InnerVolumeSpecName "kube-api-access-sxj7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:54.314714 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.314590    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-provider-credentials" pod "1943b543-d739-4742-955c-2a2f8fbc8f53" (UID: "1943b543-d739-4742-955c-2a2f8fbc8f53"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:54.424177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.423435    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:54.424177 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:54.423477    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxj7k\" (UniqueName: \"kubernetes.io/projected/1943b543-d739-4742-955c-2a2f8fbc8f53-kube-api-access-sxj7k\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:54:55.179064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:55.179026    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"
Oct 30 21:54:55.179064 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:55.179054    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25" event={"ID":"1943b543-d739-4742-955c-2a2f8fbc8f53","Type":"ContainerDied","Data":"cf307996f56e0fb4d61e5cab3c48413de5f11c25529b71da9267c7b2359963ee"}
Oct 30 21:54:55.179372 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:55.179128    5886 scope.go:122] "RemoveContainer" containerID="4e0788ed756cdb6c31a6c197775829c924596a274822d9ee54b6c8ede4f38476"
Oct 30 21:54:55.233371 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:55.233334    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"]
Oct 30 21:54:55.244022 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:55.243995    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5d3601/payload-processing-external-model-e2e-extmod-a-5d3601-584dkhw25"]
Oct 30 21:54:55.989993 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:55.989953    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1943b543-d739-4742-955c-2a2f8fbc8f53" path="/var/lib/kubelet/pods/1943b543-d739-4742-955c-2a2f8fbc8f53/volumes"
Oct 30 21:54:55.990442 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:55.990427    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6232338a-a078-4617-b66f-00423bbf1321" path="/var/lib/kubelet/pods/6232338a-a078-4617-b66f-00423bbf1321/volumes"
Oct 30 21:54:56.455528 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.455494    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"]
Oct 30 21:54:56.456415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.456394    5886 cpu_manager.go:414] "containerMap: removing container" podUID="6232338a-a078-4617-b66f-00423bbf1321" containerName="payload-processing"
Oct 30 21:54:56.456415 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.456416    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6232338a-a078-4617-b66f-00423bbf1321" containerName="payload-processing"
Oct 30 21:54:56.456547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.456460    5886 cpu_manager.go:414] "containerMap: removing container" podUID="1943b543-d739-4742-955c-2a2f8fbc8f53" containerName="payload-processing"
Oct 30 21:54:56.456547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.456466    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1943b543-d739-4742-955c-2a2f8fbc8f53" containerName="payload-processing"
Oct 30 21:54:56.456610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.456564    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="1943b543-d739-4742-955c-2a2f8fbc8f53" containerName="payload-processing"
Oct 30 21:54:56.456610 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.456576    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="6232338a-a078-4617-b66f-00423bbf1321" containerName="payload-processing"
Oct 30 21:54:56.461515 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.461491    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.480195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.480140    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"]
Oct 30 21:54:56.482103 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.482078    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"openshift-service-ca.crt\""
Oct 30 21:54:56.482103 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.482081    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c19461\""
Oct 30 21:54:56.486591 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.486548    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"kube-root-ca.crt\""
Oct 30 21:54:56.486827 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.486804    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"routing-overlay\""
Oct 30 21:54:56.515149 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.515118    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"agc-cred-055c7144178d\""
Oct 30 21:54:56.515398 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.515375    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-e2e--f2794c3d-dockercfg-xtj5d\""
Oct 30 21:54:56.570523 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.570490    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skf69\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-kube-api-access-skf69\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.570678 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.570644    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.570834 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.570813    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.570982 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.570961    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.672621 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.672585    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.672804 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.672723    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.672804 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.672789    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skf69\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-kube-api-access-skf69\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.672972 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.672872    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.673598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.673573    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.673736 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.673655    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.675675 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.675446    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.692572 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.692542    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skf69\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-kube-api-access-skf69\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:56.781541 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:56.781379    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:57.138880 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:57.138764    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"]
Oct 30 21:54:57.145132 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:54:57.145104    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcf0d56_ceb0_4741_97c0_4696b2d3c2b7.slice/crio-f60526f1fce20ffe6b9596a1ba9b5a8a8b31d58878b9e66eacd30754015dcfd2 WatchSource:0}: Error finding container f60526f1fce20ffe6b9596a1ba9b5a8a8b31d58878b9e66eacd30754015dcfd2: Status 404 returned error can't find the container with id f60526f1fce20ffe6b9596a1ba9b5a8a8b31d58878b9e66eacd30754015dcfd2
Oct 30 21:54:57.269852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:57.269773    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45" event={"ID":"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7","Type":"ContainerStarted","Data":"f60526f1fce20ffe6b9596a1ba9b5a8a8b31d58878b9e66eacd30754015dcfd2"}
Oct 30 21:54:58.041698 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.041662    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"]
Oct 30 21:54:58.045945 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.045922    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.068294 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.068264    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"]
Oct 30 21:54:58.073639 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.073615    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"agc-cred-051645226c18\""
Oct 30 21:54:58.111368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.111332    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.111368 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.111382    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.111653 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.111587    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.111728 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.111711    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67b9m\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-kube-api-access-67b9m\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.213165 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.213121    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.213695 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.213261    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67b9m\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-kube-api-access-67b9m\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.213695 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.213394    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.213695 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.213426    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.213956 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.213920    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.214126 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.214104    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.216061 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.216040    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.232562 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.232529    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67b9m\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-kube-api-access-67b9m\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.278356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.278317    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45" event={"ID":"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7","Type":"ContainerStarted","Data":"eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef"}
Oct 30 21:54:58.278494 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.278385    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:54:58.319904 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.319808    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45" podStartSLOduration=2.319796134 podStartE2EDuration="2.319796134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:58.31918788 +0000 UTC m=+2698.448439331" watchObservedRunningTime="2026-10-30 21:54:58.319796134 +0000 UTC m=+2698.449047645"
Oct 30 21:54:58.362356 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.362325    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:58.678544 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:58.678328    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"]
Oct 30 21:54:58.681279 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:54:58.681245    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25c4607_54dc_4861_bd01_d6e518a323bb.slice/crio-c09c571754d96b61469470bb8db53e98661d650022e03c8dc37334b2c02a86a4 WatchSource:0}: Error finding container c09c571754d96b61469470bb8db53e98661d650022e03c8dc37334b2c02a86a4: Status 404 returned error can't find the container with id c09c571754d96b61469470bb8db53e98661d650022e03c8dc37334b2c02a86a4
Oct 30 21:54:59.287762 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:59.287725    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm" event={"ID":"c25c4607-54dc-4861-bd01-d6e518a323bb","Type":"ContainerStarted","Data":"f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d"}
Oct 30 21:54:59.288310 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:59.287791    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm" event={"ID":"c25c4607-54dc-4861-bd01-d6e518a323bb","Type":"ContainerStarted","Data":"c09c571754d96b61469470bb8db53e98661d650022e03c8dc37334b2c02a86a4"}
Oct 30 21:54:59.288310 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:59.288096    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:54:59.329995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:54:59.329941    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm" podStartSLOduration=1.3299280470000001 podStartE2EDuration="1.329928047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:59.326689896 +0000 UTC m=+2699.455941354" watchObservedRunningTime="2026-10-30 21:54:59.329928047 +0000 UTC m=+2699.459179504"
Oct 30 21:55:09.290813 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:09.290785    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:55:10.300481 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:10.300452    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:55:10.371077 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:10.371038    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"]
Oct 30 21:55:11.048213 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:55:11.048112    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:55:15.371382 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.371338    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45" podUID="1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" containerName="payload-processing" containerID="cri-o://eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef" gracePeriod=25
Oct 30 21:55:15.666598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.666570    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:55:15.697509 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.697478    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-routing-overlay-volume\") pod \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") "
Oct 30 21:55:15.697693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.697533    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-plugins-config-volume\") pod \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") "
Oct 30 21:55:15.697693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.697585    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-provider-credentials\") pod \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") "
Oct 30 21:55:15.697693 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.697631    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-kube-api-access-skf69\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-kube-api-access-skf69\") pod \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\" (UID: \"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7\") "
Oct 30 21:55:15.697902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.697876    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-routing-overlay-volume" pod "1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" (UID: "1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:15.698031 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.698009    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-plugins-config-volume" pod "1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" (UID: "1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:15.698282 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.698263    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:55:15.698355 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.698289    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:55:15.699949 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.699927    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-provider-credentials" pod "1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" (UID: "1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:15.700016 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.699951    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-kube-api-access-skf69" pod "1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" (UID: "1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7"). InnerVolumeSpecName "kube-api-access-skf69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:15.799711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.799675    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:55:15.799711 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:15.799705    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skf69\" (UniqueName: \"kubernetes.io/projected/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7-kube-api-access-skf69\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:55:16.478650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.478616    5886 generic.go:356] "Generic (PLEG): container finished" podID="1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" containerID="eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef" exitCode=0
Oct 30 21:55:16.479105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.478662    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"
Oct 30 21:55:16.479105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.478711    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45" event={"ID":"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7","Type":"ContainerDied","Data":"eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef"}
Oct 30 21:55:16.479105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.478774    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45" event={"ID":"1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7","Type":"ContainerDied","Data":"f60526f1fce20ffe6b9596a1ba9b5a8a8b31d58878b9e66eacd30754015dcfd2"}
Oct 30 21:55:16.479105 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.478797    5886 scope.go:122] "RemoveContainer" containerID="eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef"
Oct 30 21:55:16.488857 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.488830    5886 scope.go:122] "RemoveContainer" containerID="eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef"
Oct 30 21:55:16.489142 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:55:16.489118    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef\": container with ID starting with eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef not found: ID does not exist" containerID="eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef"
Oct 30 21:55:16.489239 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.489149    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef"} err="failed to get container status \"eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef\": rpc error: code = NotFound desc = could not find container \"eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef\": container with ID starting with eb5779fc2d0e30e826c58796f3a95218a5b60d6fb518a03f886422958df6c4ef not found: ID does not exist"
Oct 30 21:55:16.534721 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.534667    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"]
Oct 30 21:55:16.541348 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:16.541296    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-645bxlm45"]
Oct 30 21:55:17.991813 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:17.991757    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" path="/var/lib/kubelet/pods/1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7/volumes"
Oct 30 21:55:33.668566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.668533    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"]
Oct 30 21:55:33.669282 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.669266    5886 cpu_manager.go:414] "containerMap: removing container" podUID="1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" containerName="payload-processing"
Oct 30 21:55:33.669340 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.669283    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" containerName="payload-processing"
Oct 30 21:55:33.669388 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.669378    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcf0d56-ceb0-4741-97c0-4696b2d3c2b7" containerName="payload-processing"
Oct 30 21:55:33.673321 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.673304    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.699013 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.698980    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"agc-cred-68bf0c456c5f\""
Oct 30 21:55:33.699225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.699059    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"e2e-cfl-3e7cf6-provider-cred\""
Oct 30 21:55:33.699225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.699136    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"agc-cred-024a2dd2dd96\""
Oct 30 21:55:33.699225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.699221    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-805edc\""
Oct 30 21:55:33.699430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.699285    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"routing-overlay\""
Oct 30 21:55:33.699430 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.699380    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"kube-root-ca.crt\""
Oct 30 21:55:33.699598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.699462    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"openshift-service-ca.crt\""
Oct 30 21:55:33.699598 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.699505    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-805edc\"/\"payload-processing-external-model-e2e--c4b58ec6-dockercfg-hxwmh\""
Oct 30 21:55:33.705285 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.705255    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"]
Oct 30 21:55:33.784950 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.784909    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.785146 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.785114    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.785241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.785169    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.785298 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.785278    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmzm\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-kube-api-access-lbmzm\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.887297 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.887252    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.887505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.887326    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.887505 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.887370    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmzm\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-kube-api-access-lbmzm\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.887656 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.887537    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.888471 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.888443    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.888825 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.888797    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.890833 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.890811    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.906139 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.906113    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmzm\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-kube-api-access-lbmzm\") pod \"payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") " pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:33.991652 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:33.991571    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:34.305868 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:34.305776    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"]
Oct 30 21:55:34.310583 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:55:34.310554    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ff8d55_919e_4b22_b602_2f7482ec018a.slice/crio-1e202225fe42870e1f6b04ddd3a12ebe2824d82063c725eff1780e5a1718cc16 WatchSource:0}: Error finding container 1e202225fe42870e1f6b04ddd3a12ebe2824d82063c725eff1780e5a1718cc16: Status 404 returned error can't find the container with id 1e202225fe42870e1f6b04ddd3a12ebe2824d82063c725eff1780e5a1718cc16
Oct 30 21:55:34.658566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:34.658530    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4" event={"ID":"31ff8d55-919e-4b22-b602-2f7482ec018a","Type":"ContainerStarted","Data":"6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66"}
Oct 30 21:55:34.658785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:34.658582    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4" event={"ID":"31ff8d55-919e-4b22-b602-2f7482ec018a","Type":"ContainerStarted","Data":"1e202225fe42870e1f6b04ddd3a12ebe2824d82063c725eff1780e5a1718cc16"}
Oct 30 21:55:34.658785 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:34.658623    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:55:34.710049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:34.709984    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4" podStartSLOduration=1.709966138 podStartE2EDuration="1.709966138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:34.707574551 +0000 UTC m=+2734.836826009" watchObservedRunningTime="2026-10-30 21:55:34.709966138 +0000 UTC m=+2734.839217596"
Oct 30 21:55:41.029628 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:55:41.029581    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:55:45.682134 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:55:45.680966    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:56:11.031790 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:56:11.031753    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:56:30.950450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:30.950418    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"]
Oct 30 21:56:30.954585 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:30.954568    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:30.995866 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:30.995834    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"]
Oct 30 21:56:31.000661 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.000618    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.000800 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.000694    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.000800 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.000747    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.000926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.000810    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.000926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.000841    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.000926 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.000890    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.001097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.000933    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.001097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.000964    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4rj\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-kube-api-access-nf4rj\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.001097 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.001002    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.102406 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.102368    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.102618 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.102475    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103044    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103103    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103138    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103241 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103233    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103282    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103318    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4rj\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-kube-api-access-nf4rj\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103446 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103393    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103604 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103519    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.103671 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.103619    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.104397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.104173    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.104397 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.104343    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.104616 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.104434    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.105200 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.105177    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.105517 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.105493    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.135352 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.135324    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.135482 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.135398    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4rj\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-kube-api-access-nf4rj\") pod \"maas-default-gateway-openshift-default-77f556d647-7bsl8\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.270017 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.269924    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:31.596117 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.595649    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"]
Oct 30 21:56:31.617962 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:56:31.617900    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab4d127_ceba_4fe7_9ff0_3f7a4d6e7b7a.slice/crio-17eec25294ec679f12ddbcf551f1e4bef269fa5222d28881f656a6cb0dfd61f5 WatchSource:0}: Error finding container 17eec25294ec679f12ddbcf551f1e4bef269fa5222d28881f656a6cb0dfd61f5: Status 404 returned error can't find the container with id 17eec25294ec679f12ddbcf551f1e4bef269fa5222d28881f656a6cb0dfd61f5
Oct 30 21:56:31.647756 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.634586    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:31.647756 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.634689    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:31.647756 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:31.634742    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:56:32.183238 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:32.183199    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" event={"ID":"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a","Type":"ContainerStarted","Data":"fce73bd2c701c040500f65fa3d5a8d6322ee0e153ef0bf424bd923d362d5d8a0"}
Oct 30 21:56:32.183754 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:32.183272    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" event={"ID":"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a","Type":"ContainerStarted","Data":"17eec25294ec679f12ddbcf551f1e4bef269fa5222d28881f656a6cb0dfd61f5"}
Oct 30 21:56:32.235035 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:32.234977    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podStartSLOduration=2.234962692 podStartE2EDuration="2.234962692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:32.234337474 +0000 UTC m=+2792.363588933" watchObservedRunningTime="2026-10-30 21:56:32.234962692 +0000 UTC m=+2792.364214149"
Oct 30 21:56:32.270970 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:32.270935    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:32.272832 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:32.272802    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7bsl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:56:32.272974 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:32.272867    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 21:56:33.274194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:33.273610    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7bsl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:56:33.274194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:33.273680    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 21:56:34.275180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:34.271533    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7bsl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:56:34.275180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:34.271611    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 21:56:35.284178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:35.273498    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7bsl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:56:35.284178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:35.273568    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 21:56:36.280574 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:36.280531    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7bsl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:56:36.281129 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:36.281099    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 21:56:37.285187 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:37.285128    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7bsl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:56:37.285786 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:37.285750    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 21:56:38.270910 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:38.270868    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7bsl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:56:38.271102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:38.270937    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 21:56:39.270747 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:39.270689    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7bsl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 21:56:39.271219 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:39.270773    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 21:56:40.748390 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:40.748314    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:40.749015 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:40.748991    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:40.749718 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:56:40.749695    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 21:56:41.032451 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:56:41.032370    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:57:02.030360 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:02.030311    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"]
Oct 30 21:57:07.031065 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.030991    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4" podUID="31ff8d55-919e-4b22-b602-2f7482ec018a" containerName="payload-processing" containerID="cri-o://6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66" gracePeriod=25
Oct 30 21:57:07.327019 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.326992    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:57:07.413927 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.413887    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-plugins-config-volume\") pod \"31ff8d55-919e-4b22-b602-2f7482ec018a\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") "
Oct 30 21:57:07.414266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.414009    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-routing-overlay-volume\") pod \"31ff8d55-919e-4b22-b602-2f7482ec018a\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") "
Oct 30 21:57:07.414266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.414083    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-provider-credentials\") pod \"31ff8d55-919e-4b22-b602-2f7482ec018a\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") "
Oct 30 21:57:07.414266 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.414232    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-kube-api-access-lbmzm\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-kube-api-access-lbmzm\") pod \"31ff8d55-919e-4b22-b602-2f7482ec018a\" (UID: \"31ff8d55-919e-4b22-b602-2f7482ec018a\") "
Oct 30 21:57:07.414525 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.414494    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-routing-overlay-volume" pod "31ff8d55-919e-4b22-b602-2f7482ec018a" (UID: "31ff8d55-919e-4b22-b602-2f7482ec018a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:07.414882 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.414853    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:07.415092 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.415069    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-plugins-config-volume" pod "31ff8d55-919e-4b22-b602-2f7482ec018a" (UID: "31ff8d55-919e-4b22-b602-2f7482ec018a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:07.416608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.416583    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-provider-credentials" pod "31ff8d55-919e-4b22-b602-2f7482ec018a" (UID: "31ff8d55-919e-4b22-b602-2f7482ec018a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:07.416608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.416600    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-kube-api-access-lbmzm" pod "31ff8d55-919e-4b22-b602-2f7482ec018a" (UID: "31ff8d55-919e-4b22-b602-2f7482ec018a"). InnerVolumeSpecName "kube-api-access-lbmzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:07.488954 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.488913    5886 generic.go:356] "Generic (PLEG): container finished" podID="31ff8d55-919e-4b22-b602-2f7482ec018a" containerID="6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66" exitCode=0
Oct 30 21:57:07.489147 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.488959    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"
Oct 30 21:57:07.489147 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.488992    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4" event={"ID":"31ff8d55-919e-4b22-b602-2f7482ec018a","Type":"ContainerDied","Data":"6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66"}
Oct 30 21:57:07.489147 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.489055    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4" event={"ID":"31ff8d55-919e-4b22-b602-2f7482ec018a","Type":"ContainerDied","Data":"1e202225fe42870e1f6b04ddd3a12ebe2824d82063c725eff1780e5a1718cc16"}
Oct 30 21:57:07.489147 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.489085    5886 scope.go:122] "RemoveContainer" containerID="6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66"
Oct 30 21:57:07.501799 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.501770    5886 scope.go:122] "RemoveContainer" containerID="6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66"
Oct 30 21:57:07.502071 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:57:07.502044    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66\": container with ID starting with 6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66 not found: ID does not exist" containerID="6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66"
Oct 30 21:57:07.502178 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.502073    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66"} err="failed to get container status \"6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66\": rpc error: code = NotFound desc = could not find container \"6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66\": container with ID starting with 6cdfbffa52ee96b65afda07fc426ce6950bc71e7cd7df61e8f04693635a03b66 not found: ID does not exist"
Oct 30 21:57:07.515907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.515882    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ff8d55-919e-4b22-b602-2f7482ec018a-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:07.515907 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.515906    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:07.516090 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.515916    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmzm\" (UniqueName: \"kubernetes.io/projected/31ff8d55-919e-4b22-b602-2f7482ec018a-kube-api-access-lbmzm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:07.537959 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.537928    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"]
Oct 30 21:57:07.544091 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.544066    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-805edc/payload-processing-external-model-e2e-extmod-a-805edc-5546zcpp4"]
Oct 30 21:57:07.990246 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:07.990214    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ff8d55-919e-4b22-b602-2f7482ec018a" path="/var/lib/kubelet/pods/31ff8d55-919e-4b22-b602-2f7482ec018a/volumes"
Oct 30 21:57:11.030582 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:57:11.030539    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:57:18.488389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.488348    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"]
Oct 30 21:57:18.490774 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.489247    5886 cpu_manager.go:414] "containerMap: removing container" podUID="31ff8d55-919e-4b22-b602-2f7482ec018a" containerName="payload-processing"
Oct 30 21:57:18.490774 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.489267    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ff8d55-919e-4b22-b602-2f7482ec018a" containerName="payload-processing"
Oct 30 21:57:18.490774 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.489400    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ff8d55-919e-4b22-b602-2f7482ec018a" containerName="payload-processing"
Oct 30 21:57:18.493327 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.493302    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.528839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.518534    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b8d814a2cf4e\""
Oct 30 21:57:18.528839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.518718    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:57:18.528839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.518785    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rxxkz\""
Oct 30 21:57:18.528839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.518827    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b35fff36b219\""
Oct 30 21:57:18.528839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.518886    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:57:18.528839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.518935    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:57:18.528839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.518992    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:57:18.531209 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.531174    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"]
Oct 30 21:57:18.541747 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.541611    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-provider-credentials\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.541747 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.541717    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-routing-overlay-volume\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.542021 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.541918    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-plugins-config-volume\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.542079 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.542039    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgsw\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-kube-api-access-bqgsw\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.643179 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.643129    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-provider-credentials\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.643393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.643244    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-routing-overlay-volume\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.643393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.643317    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-plugins-config-volume\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.643393 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.643389    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgsw\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-kube-api-access-bqgsw\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.644071 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.644035    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-routing-overlay-volume\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.644303 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.644284    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-plugins-config-volume\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.645877 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.645843    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-provider-credentials\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.663023 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.662988    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgsw\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-kube-api-access-bqgsw\") pod \"payload-processing-external-model-cf7f876d6-ft79x\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") " pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:18.809165 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:18.809053    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:19.123125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:19.123094    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"]
Oct 30 21:57:19.126486 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:57:19.126451    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e952fc6_d840_4bea_85a0_296ba6a616bd.slice/crio-1aca601c69276e350c1eb1712504dd110ff85b85c272f23d4ee3fb34b21b2352 WatchSource:0}: Error finding container 1aca601c69276e350c1eb1712504dd110ff85b85c272f23d4ee3fb34b21b2352: Status 404 returned error can't find the container with id 1aca601c69276e350c1eb1712504dd110ff85b85c272f23d4ee3fb34b21b2352
Oct 30 21:57:19.584429 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:19.584384    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x" event={"ID":"6e952fc6-d840-4bea-85a0-296ba6a616bd","Type":"ContainerStarted","Data":"c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f"}
Oct 30 21:57:19.584928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:19.584450    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:19.584928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:19.584473    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x" event={"ID":"6e952fc6-d840-4bea-85a0-296ba6a616bd","Type":"ContainerStarted","Data":"1aca601c69276e350c1eb1712504dd110ff85b85c272f23d4ee3fb34b21b2352"}
Oct 30 21:57:19.625012 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:19.624957    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x" podStartSLOduration=1.624940735 podStartE2EDuration="1.624940735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:19.621297559 +0000 UTC m=+2839.750549029" watchObservedRunningTime="2026-10-30 21:57:19.624940735 +0000 UTC m=+2839.754192192"
Oct 30 21:57:26.742910 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:26.742874    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"]
Oct 30 21:57:30.595185 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:30.595128    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:57:31.744087 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:31.744043    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm" podUID="c25c4607-54dc-4861-bd01-d6e518a323bb" containerName="payload-processing" containerID="cri-o://f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d" gracePeriod=25
Oct 30 21:57:32.288229 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.284453    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:57:32.401969 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.401940    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-provider-credentials\") pod \"c25c4607-54dc-4861-bd01-d6e518a323bb\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") "
Oct 30 21:57:32.402227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.401992    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-routing-overlay-volume\") pod \"c25c4607-54dc-4861-bd01-d6e518a323bb\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") "
Oct 30 21:57:32.402227 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.402210    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-kube-api-access-67b9m\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-kube-api-access-67b9m\") pod \"c25c4607-54dc-4861-bd01-d6e518a323bb\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") "
Oct 30 21:57:32.402475 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.402449    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-routing-overlay-volume" pod "c25c4607-54dc-4861-bd01-d6e518a323bb" (UID: "c25c4607-54dc-4861-bd01-d6e518a323bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:32.402685 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.402663    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-plugins-config-volume\") pod \"c25c4607-54dc-4861-bd01-d6e518a323bb\" (UID: \"c25c4607-54dc-4861-bd01-d6e518a323bb\") "
Oct 30 21:57:32.403235 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.403209    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-plugins-config-volume" pod "c25c4607-54dc-4861-bd01-d6e518a323bb" (UID: "c25c4607-54dc-4861-bd01-d6e518a323bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:32.403551 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.403534    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:32.403650 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.403558    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4607-54dc-4861-bd01-d6e518a323bb-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:32.405053 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.405030    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-provider-credentials" pod "c25c4607-54dc-4861-bd01-d6e518a323bb" (UID: "c25c4607-54dc-4861-bd01-d6e518a323bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:32.405501 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.405483    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-kube-api-access-67b9m" pod "c25c4607-54dc-4861-bd01-d6e518a323bb" (UID: "c25c4607-54dc-4861-bd01-d6e518a323bb"). InnerVolumeSpecName "kube-api-access-67b9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:32.504886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.504827    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:32.504886 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.504888    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67b9m\" (UniqueName: \"kubernetes.io/projected/c25c4607-54dc-4861-bd01-d6e518a323bb-kube-api-access-67b9m\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:32.687490 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.687401    5886 generic.go:356] "Generic (PLEG): container finished" podID="c25c4607-54dc-4861-bd01-d6e518a323bb" containerID="f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d" exitCode=0
Oct 30 21:57:32.687490 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.687467    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"
Oct 30 21:57:32.687712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.687495    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm" event={"ID":"c25c4607-54dc-4861-bd01-d6e518a323bb","Type":"ContainerDied","Data":"f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d"}
Oct 30 21:57:32.687712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.687566    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm" event={"ID":"c25c4607-54dc-4861-bd01-d6e518a323bb","Type":"ContainerDied","Data":"c09c571754d96b61469470bb8db53e98661d650022e03c8dc37334b2c02a86a4"}
Oct 30 21:57:32.687712 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.687603    5886 scope.go:122] "RemoveContainer" containerID="f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d"
Oct 30 21:57:32.701183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.701140    5886 scope.go:122] "RemoveContainer" containerID="f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d"
Oct 30 21:57:32.701613 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:57:32.701574    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d\": container with ID starting with f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d not found: ID does not exist" containerID="f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d"
Oct 30 21:57:32.701724 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.701623    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d"} err="failed to get container status \"f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d\": rpc error: code = NotFound desc = could not find container \"f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d\": container with ID starting with f3d41ae4a8ef7c0c56447f1f3f35be8face97c92c58fd9e399b13556a510c70d not found: ID does not exist"
Oct 30 21:57:32.746194 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.744408    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"]
Oct 30 21:57:32.750331 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:32.750306    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-59f4pglcm"]
Oct 30 21:57:34.043090 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:34.043052    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25c4607-54dc-4861-bd01-d6e518a323bb" path="/var/lib/kubelet/pods/c25c4607-54dc-4861-bd01-d6e518a323bb/volumes"
Oct 30 21:57:41.029418 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:57:41.028763    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:57:43.760287 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.760248    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"]
Oct 30 21:57:43.761271 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.761251    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c25c4607-54dc-4861-bd01-d6e518a323bb" containerName="payload-processing"
Oct 30 21:57:43.761271 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.761271    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25c4607-54dc-4861-bd01-d6e518a323bb" containerName="payload-processing"
Oct 30 21:57:43.761413 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.761373    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25c4607-54dc-4861-bd01-d6e518a323bb" containerName="payload-processing"
Oct 30 21:57:43.766547 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.766527    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.784269 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.784238    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"]
Oct 30 21:57:43.790455 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.790428    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3a9a0ef429f7\""
Oct 30 21:57:43.812402 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.812370    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-plugins-config-volume\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.812623 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.812605    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-provider-credentials\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.812688 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.812646    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-routing-overlay-volume\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.812688 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.812669    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42zx\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-kube-api-access-q42zx\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.913641 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.913606    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-provider-credentials\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.913837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.913652    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-routing-overlay-volume\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.913837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.913677    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42zx\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-kube-api-access-q42zx\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.913837 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.913767    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-plugins-config-volume\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.914556 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.914532    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-routing-overlay-volume\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.914884 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.914858    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-plugins-config-volume\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.916566 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.916542    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-provider-credentials\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:43.932220 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:43.932197    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42zx\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-kube-api-access-q42zx\") pod \"payload-processing-external-model-6bffc7c545-v4bmm\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") " pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:44.087410 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:44.087297    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:44.395608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:44.395475    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"]
Oct 30 21:57:44.398959 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:57:44.398922    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91b122e_6186_4dbb_9293_f7d124f1d81c.slice/crio-a7bbfcfa440ae4c470839d66ee72a9dc5dc4cfdd24064283326a3e6da794f72f WatchSource:0}: Error finding container a7bbfcfa440ae4c470839d66ee72a9dc5dc4cfdd24064283326a3e6da794f72f: Status 404 returned error can't find the container with id a7bbfcfa440ae4c470839d66ee72a9dc5dc4cfdd24064283326a3e6da794f72f
Oct 30 21:57:44.831467 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:44.831433    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm" event={"ID":"d91b122e-6186-4dbb-9293-f7d124f1d81c","Type":"ContainerStarted","Data":"ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2"}
Oct 30 21:57:44.831902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:44.831486    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm" event={"ID":"d91b122e-6186-4dbb-9293-f7d124f1d81c","Type":"ContainerStarted","Data":"a7bbfcfa440ae4c470839d66ee72a9dc5dc4cfdd24064283326a3e6da794f72f"}
Oct 30 21:57:44.831902 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:44.831549    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:46.150204 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.150126    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm" podStartSLOduration=3.150106033 podStartE2EDuration="3.150106033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:44.897276449 +0000 UTC m=+2865.026527907" watchObservedRunningTime="2026-10-30 21:57:46.150106033 +0000 UTC m=+2866.279357491"
Oct 30 21:57:46.150911 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.150891    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"]
Oct 30 21:57:46.156047 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.156022    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.186114 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.186085    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"]
Oct 30 21:57:46.245067 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245022    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.245263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245125    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.245263 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245183    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9cdp\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-kube-api-access-x9cdp\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.245365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245286    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.245365 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245345    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.245534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245463    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.245534 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245512    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.245659 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245645    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.245700 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.245673    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.346863 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.346827    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.346863 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.346868    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.346931    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.346955    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.346975    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9cdp\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-kube-api-access-x9cdp\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.347023    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347102 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.347080    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.347217    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.347331    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.347384    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347435 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.347415    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347627 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.347474    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.347664 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.347628    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.348135 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.348110    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.349454 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.349425    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.349564 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.349501    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.365852 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.365824    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9cdp\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-kube-api-access-x9cdp\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.366337 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.366316    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4rjnz\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.475116 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.475026    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:46.801049 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.800986    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"]
Oct 30 21:57:46.803847 ip-10-0-129-84 kubenswrapper[5886]: W1030 21:57:46.803808    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf54c5f_a5f8_421a_be0e_c75ee4906734.slice/crio-273bc98dc34fecacb2f923e57ba489b6d4eae48070e0a1c8245013334cecebe3 WatchSource:0}: Error finding container 273bc98dc34fecacb2f923e57ba489b6d4eae48070e0a1c8245013334cecebe3: Status 404 returned error can't find the container with id 273bc98dc34fecacb2f923e57ba489b6d4eae48070e0a1c8245013334cecebe3
Oct 30 21:57:46.806812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.806776    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:46.806936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.806843    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:46.806936 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.806873    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 21:57:46.849991 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:46.849962    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" event={"ID":"bbf54c5f-a5f8-421a-be0e-c75ee4906734","Type":"ContainerStarted","Data":"273bc98dc34fecacb2f923e57ba489b6d4eae48070e0a1c8245013334cecebe3"}
Oct 30 21:57:47.859056 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:47.859020    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" event={"ID":"bbf54c5f-a5f8-421a-be0e-c75ee4906734","Type":"ContainerStarted","Data":"b4e9822ffdba52e9b03cee6c6f595919839aa020bf4570f094f0d34d73122651"}
Oct 30 21:57:47.929605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:47.929549    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podStartSLOduration=1.9295338 podStartE2EDuration="1.9295338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:47.91658325 +0000 UTC m=+2868.045834697" watchObservedRunningTime="2026-10-30 21:57:47.9295338 +0000 UTC m=+2868.058785257"
Oct 30 21:57:48.490078 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:48.490041    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:48.491389 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:48.491358    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4rjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:57:48.491584 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:48.491560    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:57:48.917595 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:48.917517    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"]
Oct 30 21:57:49.479624 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:49.479582    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4rjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:57:49.479942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:49.479911    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:57:50.483180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:50.477526    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4rjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:57:50.483180 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:50.477588    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:57:51.477183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:51.476673    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4rjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:57:51.477183 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:51.476735    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:57:52.484181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:52.477502    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4rjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:57:52.484181 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:52.477563    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:57:53.475457 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:53.475420    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4rjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:57:53.475670 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:53.475487    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:57:53.918719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:53.918673    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm" podUID="d91b122e-6186-4dbb-9293-f7d124f1d81c" containerName="payload-processing" containerID="cri-o://ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2" gracePeriod=25
Oct 30 21:57:54.346928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.346901    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:54.384094 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.384057    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-routing-overlay-volume\") pod \"d91b122e-6186-4dbb-9293-f7d124f1d81c\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") "
Oct 30 21:57:54.384305 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.384206    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-kube-api-access-q42zx\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-kube-api-access-q42zx\") pod \"d91b122e-6186-4dbb-9293-f7d124f1d81c\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") "
Oct 30 21:57:54.384305 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.384253    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-provider-credentials\") pod \"d91b122e-6186-4dbb-9293-f7d124f1d81c\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") "
Oct 30 21:57:54.384305 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.384287    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-plugins-config-volume\") pod \"d91b122e-6186-4dbb-9293-f7d124f1d81c\" (UID: \"d91b122e-6186-4dbb-9293-f7d124f1d81c\") "
Oct 30 21:57:54.384516 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.384486    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-routing-overlay-volume" pod "d91b122e-6186-4dbb-9293-f7d124f1d81c" (UID: "d91b122e-6186-4dbb-9293-f7d124f1d81c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:54.384823 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.384789    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-plugins-config-volume" pod "d91b122e-6186-4dbb-9293-f7d124f1d81c" (UID: "d91b122e-6186-4dbb-9293-f7d124f1d81c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:54.384995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.384973    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:54.384995 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.384996    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91b122e-6186-4dbb-9293-f7d124f1d81c-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:54.386942 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.386866    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-kube-api-access-q42zx" pod "d91b122e-6186-4dbb-9293-f7d124f1d81c" (UID: "d91b122e-6186-4dbb-9293-f7d124f1d81c"). InnerVolumeSpecName "kube-api-access-q42zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:54.388608 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.388580    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-provider-credentials" pod "d91b122e-6186-4dbb-9293-f7d124f1d81c" (UID: "d91b122e-6186-4dbb-9293-f7d124f1d81c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:54.475839 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.475799    5886 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4rjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:57:54.476059 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.475861    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:57:54.486286 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.486201    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q42zx\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-kube-api-access-q42zx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:54.486286 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:54.486233    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d91b122e-6186-4dbb-9293-f7d124f1d81c-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:57:55.088058 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.088017    5886 generic.go:356] "Generic (PLEG): container finished" podID="d91b122e-6186-4dbb-9293-f7d124f1d81c" containerID="ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2" exitCode=0
Oct 30 21:57:55.088605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.088224    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm" event={"ID":"d91b122e-6186-4dbb-9293-f7d124f1d81c","Type":"ContainerDied","Data":"ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2"}
Oct 30 21:57:55.088605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.088276    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm" event={"ID":"d91b122e-6186-4dbb-9293-f7d124f1d81c","Type":"ContainerDied","Data":"a7bbfcfa440ae4c470839d66ee72a9dc5dc4cfdd24064283326a3e6da794f72f"}
Oct 30 21:57:55.088605 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.088306    5886 scope.go:122] "RemoveContainer" containerID="ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2"
Oct 30 21:57:55.088784 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.088615    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"
Oct 30 21:57:55.107873 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.107849    5886 scope.go:122] "RemoveContainer" containerID="ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2"
Oct 30 21:57:55.108314 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:57:55.108287    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2\": container with ID starting with ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2 not found: ID does not exist" containerID="ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2"
Oct 30 21:57:55.108439 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.108319    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2"} err="failed to get container status \"ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2\": rpc error: code = NotFound desc = could not find container \"ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2\": container with ID starting with ac4196b5796c97442969484c74730bfb00796bdc8ea9b57c087b12271b3ea1e2 not found: ID does not exist"
Oct 30 21:57:55.144536 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.144498    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"]
Oct 30 21:57:55.153195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.153135    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bffc7c545-v4bmm"]
Oct 30 21:57:55.993125 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:55.993084    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91b122e-6186-4dbb-9293-f7d124f1d81c" path="/var/lib/kubelet/pods/d91b122e-6186-4dbb-9293-f7d124f1d81c/volumes"
Oct 30 21:57:56.136421 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:56.136301    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:56.136928 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:56.136792    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:57:56.138044 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:57:56.138019    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 21:58:11.030336 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:58:11.030244    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:58:26.084624 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:26.084586    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"]
Oct 30 21:58:31.085212 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.085133    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x" podUID="6e952fc6-d840-4bea-85a0-296ba6a616bd" containerName="payload-processing" containerID="cri-o://c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f" gracePeriod=25
Oct 30 21:58:31.370354 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.370329    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:58:31.480214 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.480175    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-routing-overlay-volume\") pod \"6e952fc6-d840-4bea-85a0-296ba6a616bd\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") "
Oct 30 21:58:31.480423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.480361    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-provider-credentials\") pod \"6e952fc6-d840-4bea-85a0-296ba6a616bd\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") "
Oct 30 21:58:31.480423 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.480398    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-kube-api-access-bqgsw\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-kube-api-access-bqgsw\") pod \"6e952fc6-d840-4bea-85a0-296ba6a616bd\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") "
Oct 30 21:58:31.480543 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.480470    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-plugins-config-volume\") pod \"6e952fc6-d840-4bea-85a0-296ba6a616bd\" (UID: \"6e952fc6-d840-4bea-85a0-296ba6a616bd\") "
Oct 30 21:58:31.480625 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.480595    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-routing-overlay-volume" pod "6e952fc6-d840-4bea-85a0-296ba6a616bd" (UID: "6e952fc6-d840-4bea-85a0-296ba6a616bd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:31.481006 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.480980    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-plugins-config-volume" pod "6e952fc6-d840-4bea-85a0-296ba6a616bd" (UID: "6e952fc6-d840-4bea-85a0-296ba6a616bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:31.481195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.481130    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:58:31.481195 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.481150    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e952fc6-d840-4bea-85a0-296ba6a616bd-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:58:31.482719 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.482692    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-kube-api-access-bqgsw" pod "6e952fc6-d840-4bea-85a0-296ba6a616bd" (UID: "6e952fc6-d840-4bea-85a0-296ba6a616bd"). InnerVolumeSpecName "kube-api-access-bqgsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:31.482943 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.482926    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-provider-credentials" pod "6e952fc6-d840-4bea-85a0-296ba6a616bd" (UID: "6e952fc6-d840-4bea-85a0-296ba6a616bd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:31.582876 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.582818    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgsw\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-kube-api-access-bqgsw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:58:31.582876 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.582874    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e952fc6-d840-4bea-85a0-296ba6a616bd-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 21:58:31.828676 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.828642    5886 generic.go:356] "Generic (PLEG): container finished" podID="6e952fc6-d840-4bea-85a0-296ba6a616bd" containerID="c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f" exitCode=0
Oct 30 21:58:31.828864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.828692    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"
Oct 30 21:58:31.828864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.828725    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x" event={"ID":"6e952fc6-d840-4bea-85a0-296ba6a616bd","Type":"ContainerDied","Data":"c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f"}
Oct 30 21:58:31.828864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.828773    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x" event={"ID":"6e952fc6-d840-4bea-85a0-296ba6a616bd","Type":"ContainerDied","Data":"1aca601c69276e350c1eb1712504dd110ff85b85c272f23d4ee3fb34b21b2352"}
Oct 30 21:58:31.828864 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.828794    5886 scope.go:122] "RemoveContainer" containerID="c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f"
Oct 30 21:58:31.840812 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.840792    5886 scope.go:122] "RemoveContainer" containerID="c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f"
Oct 30 21:58:31.841123 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:58:31.841102    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f\": container with ID starting with c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f not found: ID does not exist" containerID="c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f"
Oct 30 21:58:31.841225 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.841129    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f"} err="failed to get container status \"c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f\": rpc error: code = NotFound desc = could not find container \"c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f\": container with ID starting with c0f567c40fc0757c38b052af406b84a0013df33aa45f3634164684c5717ee97f not found: ID does not exist"
Oct 30 21:58:31.880835 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.880807    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"]
Oct 30 21:58:31.884450 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.884425    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cf7f876d6-ft79x"]
Oct 30 21:58:31.989492 ip-10-0-129-84 kubenswrapper[5886]: I1030 21:58:31.989458    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e952fc6-d840-4bea-85a0-296ba6a616bd" path="/var/lib/kubelet/pods/6e952fc6-d840-4bea-85a0-296ba6a616bd/volumes"
Oct 30 21:58:41.030923 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:58:41.030875    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:59:11.030647 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:59:11.030598    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 21:59:41.062893 ip-10-0-129-84 kubenswrapper[5886]: E1030 21:59:41.062799    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:00:00.216060 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.216027    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"]
Oct 30 22:00:00.217536 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.217509    5886 cpu_manager.go:414] "containerMap: removing container" podUID="6e952fc6-d840-4bea-85a0-296ba6a616bd" containerName="payload-processing"
Oct 30 22:00:00.218826 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.218805    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e952fc6-d840-4bea-85a0-296ba6a616bd" containerName="payload-processing"
Oct 30 22:00:00.218943 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.218873    5886 cpu_manager.go:414] "containerMap: removing container" podUID="d91b122e-6186-4dbb-9293-f7d124f1d81c" containerName="payload-processing"
Oct 30 22:00:00.218943 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.218879    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91b122e-6186-4dbb-9293-f7d124f1d81c" containerName="payload-processing"
Oct 30 22:00:00.219036 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.219020    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e952fc6-d840-4bea-85a0-296ba6a616bd" containerName="payload-processing"
Oct 30 22:00:00.219079 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.219036    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91b122e-6186-4dbb-9293-f7d124f1d81c" containerName="payload-processing"
Oct 30 22:00:00.222782 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.222755    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"]
Oct 30 22:00:00.224212 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.224189    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:00.228893 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.228874    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:00.241214 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.241184    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"]
Oct 30 22:00:00.255242 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.255210    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"]
Oct 30 22:00:00.259279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.259256    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"]
Oct 30 22:00:00.264568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.264305    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:00.274056 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.274028    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:00:00.274765 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.274743    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"]
Oct 30 22:00:00.282746 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.282645    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:00.288318 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.288299    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"]
Oct 30 22:00:00.294549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.294525    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"]
Oct 30 22:00:00.312542 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.302376    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:00.312542 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.304001    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"]
Oct 30 22:00:00.312773 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.312542    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"]
Oct 30 22:00:00.325899 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.324892    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8hrv\" (UniqueName: \"kubernetes.io/projected/7818ff04-42f0-4d73-90bc-10088989e3a2-kube-api-access-w8hrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt\" (UID: \"7818ff04-42f0-4d73-90bc-10088989e3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:00.325899 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.324949    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd355a55-0491-4cab-97eb-c18e91a5deec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th\" (UID: \"dd355a55-0491-4cab-97eb-c18e91a5deec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:00.325899 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.325056    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5sfd\" (UniqueName: \"kubernetes.io/projected/dd355a55-0491-4cab-97eb-c18e91a5deec-kube-api-access-j5sfd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th\" (UID: \"dd355a55-0491-4cab-97eb-c18e91a5deec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:00.325899 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.325095    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7818ff04-42f0-4d73-90bc-10088989e3a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt\" (UID: \"7818ff04-42f0-4d73-90bc-10088989e3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:00.428937 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.428893    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44f452c-1628-487d-9edd-2b610efbdea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb\" (UID: \"b44f452c-1628-487d-9edd-2b610efbdea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:00.429226 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429022    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq\" (UID: \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:00.429226 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429098    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f720928e-4b4f-438b-b00c-95ee8d3e1141-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t\" (UID: \"f720928e-4b4f-438b-b00c-95ee8d3e1141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:00.429363 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429202    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8hrv\" (UniqueName: \"kubernetes.io/projected/7818ff04-42f0-4d73-90bc-10088989e3a2-kube-api-access-w8hrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt\" (UID: \"7818ff04-42f0-4d73-90bc-10088989e3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:00.429417 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429402    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd355a55-0491-4cab-97eb-c18e91a5deec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th\" (UID: \"dd355a55-0491-4cab-97eb-c18e91a5deec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:00.429640 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429474    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwjw\" (UniqueName: \"kubernetes.io/projected/f720928e-4b4f-438b-b00c-95ee8d3e1141-kube-api-access-tfwjw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t\" (UID: \"f720928e-4b4f-438b-b00c-95ee8d3e1141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:00.429722 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429706    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2rk\" (UniqueName: \"kubernetes.io/projected/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-kube-api-access-7n2rk\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq\" (UID: \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:00.429776 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429760    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5sfd\" (UniqueName: \"kubernetes.io/projected/dd355a55-0491-4cab-97eb-c18e91a5deec-kube-api-access-j5sfd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th\" (UID: \"dd355a55-0491-4cab-97eb-c18e91a5deec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:00.429827 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429807    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7818ff04-42f0-4d73-90bc-10088989e3a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt\" (UID: \"7818ff04-42f0-4d73-90bc-10088989e3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:00.429877 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.429845    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjth\" (UniqueName: \"kubernetes.io/projected/b44f452c-1628-487d-9edd-2b610efbdea9-kube-api-access-ncjth\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb\" (UID: \"b44f452c-1628-487d-9edd-2b610efbdea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:00.443178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.430934    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd355a55-0491-4cab-97eb-c18e91a5deec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th\" (UID: \"dd355a55-0491-4cab-97eb-c18e91a5deec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:00.443178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.432343    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7818ff04-42f0-4d73-90bc-10088989e3a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt\" (UID: \"7818ff04-42f0-4d73-90bc-10088989e3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:00.452404 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.452364    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5sfd\" (UniqueName: \"kubernetes.io/projected/dd355a55-0491-4cab-97eb-c18e91a5deec-kube-api-access-j5sfd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th\" (UID: \"dd355a55-0491-4cab-97eb-c18e91a5deec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:00.452633 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.452607    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8hrv\" (UniqueName: \"kubernetes.io/projected/7818ff04-42f0-4d73-90bc-10088989e3a2-kube-api-access-w8hrv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt\" (UID: \"7818ff04-42f0-4d73-90bc-10088989e3a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:00.532482 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.532377    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq\" (UID: \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:00.532670 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.532539    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f720928e-4b4f-438b-b00c-95ee8d3e1141-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t\" (UID: \"f720928e-4b4f-438b-b00c-95ee8d3e1141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:00.532746 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.532695    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwjw\" (UniqueName: \"kubernetes.io/projected/f720928e-4b4f-438b-b00c-95ee8d3e1141-kube-api-access-tfwjw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t\" (UID: \"f720928e-4b4f-438b-b00c-95ee8d3e1141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:00.532803 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.532764    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2rk\" (UniqueName: \"kubernetes.io/projected/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-kube-api-access-7n2rk\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq\" (UID: \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:00.532862 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.532825    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjth\" (UniqueName: \"kubernetes.io/projected/b44f452c-1628-487d-9edd-2b610efbdea9-kube-api-access-ncjth\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb\" (UID: \"b44f452c-1628-487d-9edd-2b610efbdea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:00.532925 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.532875    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44f452c-1628-487d-9edd-2b610efbdea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb\" (UID: \"b44f452c-1628-487d-9edd-2b610efbdea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:00.533195 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.533170    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq\" (UID: \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:00.533302 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.533282    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f720928e-4b4f-438b-b00c-95ee8d3e1141-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t\" (UID: \"f720928e-4b4f-438b-b00c-95ee8d3e1141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:00.533617 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.533598    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44f452c-1628-487d-9edd-2b610efbdea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb\" (UID: \"b44f452c-1628-487d-9edd-2b610efbdea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:00.547574 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.547552    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:00.553178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.552723    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwjw\" (UniqueName: \"kubernetes.io/projected/f720928e-4b4f-438b-b00c-95ee8d3e1141-kube-api-access-tfwjw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t\" (UID: \"f720928e-4b4f-438b-b00c-95ee8d3e1141\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:00.553178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.552922    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2rk\" (UniqueName: \"kubernetes.io/projected/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-kube-api-access-7n2rk\") pod \"maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq\" (UID: \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:00.554204 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.554151    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjth\" (UniqueName: \"kubernetes.io/projected/b44f452c-1628-487d-9edd-2b610efbdea9-kube-api-access-ncjth\") pod \"maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb\" (UID: \"b44f452c-1628-487d-9edd-2b610efbdea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:00.559126 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.559102    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:00.636181 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.629764    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:00.650235 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.641663    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:00.650235 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.645018    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:00.952245 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:00.952038    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"]
Oct 30 22:00:01.053083 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.051928    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"]
Oct 30 22:00:01.056527 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:00:01.056495    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44f452c_1628_487d_9edd_2b610efbdea9.slice/crio-52714b262b9e0bf476c0e74c08f49f68ff6b1601b5c01f4b280ce4a7fa9e3746 WatchSource:0}: Error finding container 52714b262b9e0bf476c0e74c08f49f68ff6b1601b5c01f4b280ce4a7fa9e3746: Status 404 returned error can't find the container with id 52714b262b9e0bf476c0e74c08f49f68ff6b1601b5c01f4b280ce4a7fa9e3746
Oct 30 22:00:01.060873 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.060772    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"]
Oct 30 22:00:01.064810 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:00:01.064783    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf720928e_4b4f_438b_b00c_95ee8d3e1141.slice/crio-b9c6bcc13d12eb59a9798e32af3aa7f2bdbef1a594c11549f6538862d4d1dce7 WatchSource:0}: Error finding container b9c6bcc13d12eb59a9798e32af3aa7f2bdbef1a594c11549f6538862d4d1dce7: Status 404 returned error can't find the container with id b9c6bcc13d12eb59a9798e32af3aa7f2bdbef1a594c11549f6538862d4d1dce7
Oct 30 22:00:01.078785 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.078763    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"]
Oct 30 22:00:01.160560 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.160535    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"]
Oct 30 22:00:01.689761 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.689731    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" event={"ID":"7818ff04-42f0-4d73-90bc-10088989e3a2","Type":"ContainerStarted","Data":"de9d6ccbcc4611f59afa6e4cd8601dc3b34cf60e756143a1922c22b39f1ed831"}
Oct 30 22:00:01.691532 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.691500    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" event={"ID":"f720928e-4b4f-438b-b00c-95ee8d3e1141","Type":"ContainerStarted","Data":"dad30af19f787a9bc112694ed12cb673eae624d81fecf0bdbf6ace69ca549552"}
Oct 30 22:00:01.691682 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.691558    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" event={"ID":"f720928e-4b4f-438b-b00c-95ee8d3e1141","Type":"ContainerStarted","Data":"b9c6bcc13d12eb59a9798e32af3aa7f2bdbef1a594c11549f6538862d4d1dce7"}
Oct 30 22:00:01.693133 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.693103    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" event={"ID":"fee86e65-5484-4a36-be26-cbc9ccf8c7ab","Type":"ContainerStarted","Data":"176e18695b604622c077b7431acc2cb320e75e3c5c99675b07df854e16ff98ff"}
Oct 30 22:00:01.693266 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.693151    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" event={"ID":"fee86e65-5484-4a36-be26-cbc9ccf8c7ab","Type":"ContainerStarted","Data":"8f5463ab9f8bb33b36c8039ee00f4f0f7d1a62b8180d48558fd17f11c27cbfd3"}
Oct 30 22:00:01.694766 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.694741    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" event={"ID":"dd355a55-0491-4cab-97eb-c18e91a5deec","Type":"ContainerStarted","Data":"c3f8f069509e033374e4d26570154e21aa8ff98489aee9f54b40ecfbcd7b3749"}
Oct 30 22:00:01.694866 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.694786    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" event={"ID":"dd355a55-0491-4cab-97eb-c18e91a5deec","Type":"ContainerStarted","Data":"49d0cfd20f784dfb314f9324fe1d1ed7de90e536e96aae44619bdfa9b737396a"}
Oct 30 22:00:01.696411 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.696380    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" event={"ID":"b44f452c-1628-487d-9edd-2b610efbdea9","Type":"ContainerStarted","Data":"0bc4f48cfb71f34f59940391d9f7e843271d1ab6548076ef418c04bb99d7e254"}
Oct 30 22:00:01.696519 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.696429    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" event={"ID":"b44f452c-1628-487d-9edd-2b610efbdea9","Type":"ContainerStarted","Data":"52714b262b9e0bf476c0e74c08f49f68ff6b1601b5c01f4b280ce4a7fa9e3746"}
Oct 30 22:00:01.729427 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.729384    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" podStartSLOduration=1.729371092 podStartE2EDuration="1.729371092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.726937531 +0000 UTC m=+3001.856188988" watchObservedRunningTime="2026-10-30 22:00:01.729371092 +0000 UTC m=+3001.858622548"
Oct 30 22:00:01.766898 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.766847    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" podStartSLOduration=1.766829946 podStartE2EDuration="1.766829946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.766630064 +0000 UTC m=+3001.895881522" watchObservedRunningTime="2026-10-30 22:00:01.766829946 +0000 UTC m=+3001.896081406"
Oct 30 22:00:01.803599 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.803550    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" podStartSLOduration=1.803535429 podStartE2EDuration="1.803535429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.802226583 +0000 UTC m=+3001.931478036" watchObservedRunningTime="2026-10-30 22:00:01.803535429 +0000 UTC m=+3001.932786903"
Oct 30 22:00:01.841917 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:01.841873    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" podStartSLOduration=1.841859796 podStartE2EDuration="1.841859796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.837252013 +0000 UTC m=+3001.966503470" watchObservedRunningTime="2026-10-30 22:00:01.841859796 +0000 UTC m=+3001.971111638"
Oct 30 22:00:02.706233 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.706192    5886 generic.go:356] "Generic (PLEG): container finished" podID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerID="dad30af19f787a9bc112694ed12cb673eae624d81fecf0bdbf6ace69ca549552" exitCode=60
Oct 30 22:00:02.706744 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.706284    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" event={"ID":"f720928e-4b4f-438b-b00c-95ee8d3e1141","Type":"ContainerDied","Data":"dad30af19f787a9bc112694ed12cb673eae624d81fecf0bdbf6ace69ca549552"}
Oct 30 22:00:02.706744 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.706569    5886 scope.go:122] "RemoveContainer" containerID="dad30af19f787a9bc112694ed12cb673eae624d81fecf0bdbf6ace69ca549552"
Oct 30 22:00:02.708378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.708349    5886 generic.go:356] "Generic (PLEG): container finished" podID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerID="176e18695b604622c077b7431acc2cb320e75e3c5c99675b07df854e16ff98ff" exitCode=60
Oct 30 22:00:02.708528 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.708474    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" event={"ID":"fee86e65-5484-4a36-be26-cbc9ccf8c7ab","Type":"ContainerDied","Data":"176e18695b604622c077b7431acc2cb320e75e3c5c99675b07df854e16ff98ff"}
Oct 30 22:00:02.708695 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.708675    5886 scope.go:122] "RemoveContainer" containerID="176e18695b604622c077b7431acc2cb320e75e3c5c99675b07df854e16ff98ff"
Oct 30 22:00:02.710398 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.710336    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" event={"ID":"dd355a55-0491-4cab-97eb-c18e91a5deec","Type":"ContainerDied","Data":"c3f8f069509e033374e4d26570154e21aa8ff98489aee9f54b40ecfbcd7b3749"}
Oct 30 22:00:02.710512 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.710264    5886 generic.go:356] "Generic (PLEG): container finished" podID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerID="c3f8f069509e033374e4d26570154e21aa8ff98489aee9f54b40ecfbcd7b3749" exitCode=60
Oct 30 22:00:02.710633 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.710599    5886 scope.go:122] "RemoveContainer" containerID="c3f8f069509e033374e4d26570154e21aa8ff98489aee9f54b40ecfbcd7b3749"
Oct 30 22:00:02.714776 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.714742    5886 generic.go:356] "Generic (PLEG): container finished" podID="b44f452c-1628-487d-9edd-2b610efbdea9" containerID="0bc4f48cfb71f34f59940391d9f7e843271d1ab6548076ef418c04bb99d7e254" exitCode=60
Oct 30 22:00:02.714870 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.714793    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" event={"ID":"b44f452c-1628-487d-9edd-2b610efbdea9","Type":"ContainerDied","Data":"0bc4f48cfb71f34f59940391d9f7e843271d1ab6548076ef418c04bb99d7e254"}
Oct 30 22:00:02.715010 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.714992    5886 scope.go:122] "RemoveContainer" containerID="0bc4f48cfb71f34f59940391d9f7e843271d1ab6548076ef418c04bb99d7e254"
Oct 30 22:00:02.721537 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.721514    5886 generic.go:356] "Generic (PLEG): container finished" podID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerID="6ca2dd3e819e40c96212a7de3ce42fe623d3a3fb0a19c33687c33ca3ee1f579e" exitCode=60
Oct 30 22:00:02.721736 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.721711    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" event={"ID":"7818ff04-42f0-4d73-90bc-10088989e3a2","Type":"ContainerDied","Data":"6ca2dd3e819e40c96212a7de3ce42fe623d3a3fb0a19c33687c33ca3ee1f579e"}
Oct 30 22:00:02.721849 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:02.721832    5886 scope.go:122] "RemoveContainer" containerID="6ca2dd3e819e40c96212a7de3ce42fe623d3a3fb0a19c33687c33ca3ee1f579e"
Oct 30 22:00:03.731194 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:03.731142    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" event={"ID":"7818ff04-42f0-4d73-90bc-10088989e3a2","Type":"ContainerStarted","Data":"140477bbba26c1f02a7fe50d0525560ca56fcb03cbfe05fda5ce4966ee74e0a2"}
Oct 30 22:00:03.733426 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:03.733398    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" event={"ID":"f720928e-4b4f-438b-b00c-95ee8d3e1141","Type":"ContainerStarted","Data":"8922c95346d27140bcca6c87f23775c023d5695842dc88a17b2cb5643c83d2ff"}
Oct 30 22:00:03.735404 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:03.735370    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" event={"ID":"fee86e65-5484-4a36-be26-cbc9ccf8c7ab","Type":"ContainerStarted","Data":"2247bca03dfac1775169845509c6194f9e880eeadc20cd73ad262df5fb5a065b"}
Oct 30 22:00:03.737301 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:03.737277    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" event={"ID":"dd355a55-0491-4cab-97eb-c18e91a5deec","Type":"ContainerStarted","Data":"b1aac802783381f468cf5029d55ddc3dea49d032b04102beeeddaa97b9450b11"}
Oct 30 22:00:03.739203 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:03.739178    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" event={"ID":"b44f452c-1628-487d-9edd-2b610efbdea9","Type":"ContainerStarted","Data":"6b3ee3d3ef1a5aed13f97c008e1e11b79a9b02a36e3aac0ee1dd32fac3a41d46"}
Oct 30 22:00:03.772395 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:03.772343    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" podStartSLOduration=3.772327491 podStartE2EDuration="3.772327491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.767115416 +0000 UTC m=+3003.896366867" watchObservedRunningTime="2026-10-30 22:00:03.772327491 +0000 UTC m=+3003.901578948"
Oct 30 22:00:04.749471 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.749430    5886 generic.go:356] "Generic (PLEG): container finished" podID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerID="140477bbba26c1f02a7fe50d0525560ca56fcb03cbfe05fda5ce4966ee74e0a2" exitCode=60
Oct 30 22:00:04.749979 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.749511    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" event={"ID":"7818ff04-42f0-4d73-90bc-10088989e3a2","Type":"ContainerDied","Data":"140477bbba26c1f02a7fe50d0525560ca56fcb03cbfe05fda5ce4966ee74e0a2"}
Oct 30 22:00:04.749979 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.749592    5886 scope.go:122] "RemoveContainer" containerID="6ca2dd3e819e40c96212a7de3ce42fe623d3a3fb0a19c33687c33ca3ee1f579e"
Oct 30 22:00:04.749979 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.749915    5886 scope.go:122] "RemoveContainer" containerID="140477bbba26c1f02a7fe50d0525560ca56fcb03cbfe05fda5ce4966ee74e0a2"
Oct 30 22:00:04.750384 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:04.750346    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt_odh-ai-gateway-infra(7818ff04-42f0-4d73-90bc-10088989e3a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2"
Oct 30 22:00:04.752097 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.752072    5886 generic.go:356] "Generic (PLEG): container finished" podID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerID="8922c95346d27140bcca6c87f23775c023d5695842dc88a17b2cb5643c83d2ff" exitCode=60
Oct 30 22:00:04.752208 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.752120    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" event={"ID":"f720928e-4b4f-438b-b00c-95ee8d3e1141","Type":"ContainerDied","Data":"8922c95346d27140bcca6c87f23775c023d5695842dc88a17b2cb5643c83d2ff"}
Oct 30 22:00:04.752511 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.752481    5886 scope.go:122] "RemoveContainer" containerID="8922c95346d27140bcca6c87f23775c023d5695842dc88a17b2cb5643c83d2ff"
Oct 30 22:00:04.752856 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:04.752831    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t_odh-ai-gateway-infra(f720928e-4b4f-438b-b00c-95ee8d3e1141)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141"
Oct 30 22:00:04.754331 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.754307    5886 generic.go:356] "Generic (PLEG): container finished" podID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerID="2247bca03dfac1775169845509c6194f9e880eeadc20cd73ad262df5fb5a065b" exitCode=60
Oct 30 22:00:04.754452 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.754397    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" event={"ID":"fee86e65-5484-4a36-be26-cbc9ccf8c7ab","Type":"ContainerDied","Data":"2247bca03dfac1775169845509c6194f9e880eeadc20cd73ad262df5fb5a065b"}
Oct 30 22:00:04.754704 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.754683    5886 scope.go:122] "RemoveContainer" containerID="2247bca03dfac1775169845509c6194f9e880eeadc20cd73ad262df5fb5a065b"
Oct 30 22:00:04.755054 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:04.755033    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq_odh-ai-gateway-infra(fee86e65-5484-4a36-be26-cbc9ccf8c7ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab"
Oct 30 22:00:04.757074 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.757049    5886 generic.go:356] "Generic (PLEG): container finished" podID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerID="b1aac802783381f468cf5029d55ddc3dea49d032b04102beeeddaa97b9450b11" exitCode=60
Oct 30 22:00:04.757214 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.757129    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" event={"ID":"dd355a55-0491-4cab-97eb-c18e91a5deec","Type":"ContainerDied","Data":"b1aac802783381f468cf5029d55ddc3dea49d032b04102beeeddaa97b9450b11"}
Oct 30 22:00:04.757441 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.757425    5886 scope.go:122] "RemoveContainer" containerID="b1aac802783381f468cf5029d55ddc3dea49d032b04102beeeddaa97b9450b11"
Oct 30 22:00:04.757856 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:04.757832    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th_odh-ai-gateway-infra(dd355a55-0491-4cab-97eb-c18e91a5deec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec"
Oct 30 22:00:04.759322 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.759302    5886 generic.go:356] "Generic (PLEG): container finished" podID="b44f452c-1628-487d-9edd-2b610efbdea9" containerID="6b3ee3d3ef1a5aed13f97c008e1e11b79a9b02a36e3aac0ee1dd32fac3a41d46" exitCode=60
Oct 30 22:00:04.759436 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.759382    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" event={"ID":"b44f452c-1628-487d-9edd-2b610efbdea9","Type":"ContainerDied","Data":"6b3ee3d3ef1a5aed13f97c008e1e11b79a9b02a36e3aac0ee1dd32fac3a41d46"}
Oct 30 22:00:04.759608 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.759590    5886 scope.go:122] "RemoveContainer" containerID="6b3ee3d3ef1a5aed13f97c008e1e11b79a9b02a36e3aac0ee1dd32fac3a41d46"
Oct 30 22:00:04.759844 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:04.759824    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb_odh-ai-gateway-infra(b44f452c-1628-487d-9edd-2b610efbdea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" podUID="b44f452c-1628-487d-9edd-2b610efbdea9"
Oct 30 22:00:04.766025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.766009    5886 scope.go:122] "RemoveContainer" containerID="dad30af19f787a9bc112694ed12cb673eae624d81fecf0bdbf6ace69ca549552"
Oct 30 22:00:04.783224 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.783201    5886 scope.go:122] "RemoveContainer" containerID="176e18695b604622c077b7431acc2cb320e75e3c5c99675b07df854e16ff98ff"
Oct 30 22:00:04.797364 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.797343    5886 scope.go:122] "RemoveContainer" containerID="c3f8f069509e033374e4d26570154e21aa8ff98489aee9f54b40ecfbcd7b3749"
Oct 30 22:00:04.812928 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:04.812895    5886 scope.go:122] "RemoveContainer" containerID="0bc4f48cfb71f34f59940391d9f7e843271d1ab6548076ef418c04bb99d7e254"
Oct 30 22:00:05.768119 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:05.768087    5886 scope.go:122] "RemoveContainer" containerID="b1aac802783381f468cf5029d55ddc3dea49d032b04102beeeddaa97b9450b11"
Oct 30 22:00:05.768719 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:05.768537    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th_odh-ai-gateway-infra(dd355a55-0491-4cab-97eb-c18e91a5deec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec"
Oct 30 22:00:05.769987 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:05.769961    5886 scope.go:122] "RemoveContainer" containerID="6b3ee3d3ef1a5aed13f97c008e1e11b79a9b02a36e3aac0ee1dd32fac3a41d46"
Oct 30 22:00:05.770366 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:05.770335    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb_odh-ai-gateway-infra(b44f452c-1628-487d-9edd-2b610efbdea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" podUID="b44f452c-1628-487d-9edd-2b610efbdea9"
Oct 30 22:00:05.771828 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:05.771807    5886 scope.go:122] "RemoveContainer" containerID="140477bbba26c1f02a7fe50d0525560ca56fcb03cbfe05fda5ce4966ee74e0a2"
Oct 30 22:00:05.772151 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:05.772126    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt_odh-ai-gateway-infra(7818ff04-42f0-4d73-90bc-10088989e3a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2"
Oct 30 22:00:05.773716 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:05.773696    5886 scope.go:122] "RemoveContainer" containerID="8922c95346d27140bcca6c87f23775c023d5695842dc88a17b2cb5643c83d2ff"
Oct 30 22:00:05.774017 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:05.773995    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t_odh-ai-gateway-infra(f720928e-4b4f-438b-b00c-95ee8d3e1141)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141"
Oct 30 22:00:05.775626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:05.775606    5886 scope.go:122] "RemoveContainer" containerID="2247bca03dfac1775169845509c6194f9e880eeadc20cd73ad262df5fb5a065b"
Oct 30 22:00:05.775918 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:05.775900    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq_odh-ai-gateway-infra(fee86e65-5484-4a36-be26-cbc9ccf8c7ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab"
Oct 30 22:00:11.029237 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:11.029196    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:00:13.981785 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:13.981749    5886 scope.go:122] "RemoveContainer" containerID="6b3ee3d3ef1a5aed13f97c008e1e11b79a9b02a36e3aac0ee1dd32fac3a41d46"
Oct 30 22:00:14.859726 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:14.857990    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" event={"ID":"b44f452c-1628-487d-9edd-2b610efbdea9","Type":"ContainerStarted","Data":"d5716b03aa3ef212880cca1012f67c498035300a46b2fe487667a427bcf77865"}
Oct 30 22:00:15.869802 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:15.869764    5886 generic.go:356] "Generic (PLEG): container finished" podID="b44f452c-1628-487d-9edd-2b610efbdea9" containerID="d5716b03aa3ef212880cca1012f67c498035300a46b2fe487667a427bcf77865" exitCode=60
Oct 30 22:00:15.870348 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:15.869804    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" event={"ID":"b44f452c-1628-487d-9edd-2b610efbdea9","Type":"ContainerDied","Data":"d5716b03aa3ef212880cca1012f67c498035300a46b2fe487667a427bcf77865"}
Oct 30 22:00:15.870348 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:15.869880    5886 scope.go:122] "RemoveContainer" containerID="6b3ee3d3ef1a5aed13f97c008e1e11b79a9b02a36e3aac0ee1dd32fac3a41d46"
Oct 30 22:00:15.870348 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:15.870342    5886 scope.go:122] "RemoveContainer" containerID="d5716b03aa3ef212880cca1012f67c498035300a46b2fe487667a427bcf77865"
Oct 30 22:00:15.870789 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:15.870762    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb_odh-ai-gateway-infra(b44f452c-1628-487d-9edd-2b610efbdea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" podUID="b44f452c-1628-487d-9edd-2b610efbdea9"
Oct 30 22:00:15.911741 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:15.911692    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"]
Oct 30 22:00:16.981093 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:16.981058    5886 scope.go:122] "RemoveContainer" containerID="140477bbba26c1f02a7fe50d0525560ca56fcb03cbfe05fda5ce4966ee74e0a2"
Oct 30 22:00:16.982678 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:16.981976    5886 scope.go:122] "RemoveContainer" containerID="2247bca03dfac1775169845509c6194f9e880eeadc20cd73ad262df5fb5a065b"
Oct 30 22:00:16.982678 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:16.982078    5886 scope.go:122] "RemoveContainer" containerID="b1aac802783381f468cf5029d55ddc3dea49d032b04102beeeddaa97b9450b11"
Oct 30 22:00:17.061073 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.061052    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:17.114065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.114031    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44f452c-1628-487d-9edd-2b610efbdea9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44f452c-1628-487d-9edd-2b610efbdea9-maas-api-service-ca\") pod \"b44f452c-1628-487d-9edd-2b610efbdea9\" (UID: \"b44f452c-1628-487d-9edd-2b610efbdea9\") "
Oct 30 22:00:17.114264 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.114245    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44f452c-1628-487d-9edd-2b610efbdea9-kube-api-access-ncjth\" (UniqueName: \"kubernetes.io/projected/b44f452c-1628-487d-9edd-2b610efbdea9-kube-api-access-ncjth\") pod \"b44f452c-1628-487d-9edd-2b610efbdea9\" (UID: \"b44f452c-1628-487d-9edd-2b610efbdea9\") "
Oct 30 22:00:17.114361 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.114339    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44f452c-1628-487d-9edd-2b610efbdea9-maas-api-service-ca" pod "b44f452c-1628-487d-9edd-2b610efbdea9" (UID: "b44f452c-1628-487d-9edd-2b610efbdea9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:17.115677 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.115653    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b44f452c-1628-487d-9edd-2b610efbdea9-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:17.117135 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.117109    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44f452c-1628-487d-9edd-2b610efbdea9-kube-api-access-ncjth" pod "b44f452c-1628-487d-9edd-2b610efbdea9" (UID: "b44f452c-1628-487d-9edd-2b610efbdea9"). InnerVolumeSpecName "kube-api-access-ncjth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:17.216988 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.216946    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncjth\" (UniqueName: \"kubernetes.io/projected/b44f452c-1628-487d-9edd-2b610efbdea9-kube-api-access-ncjth\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:17.891720 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.891622    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" event={"ID":"fee86e65-5484-4a36-be26-cbc9ccf8c7ab","Type":"ContainerStarted","Data":"fcf3e9fb2f65d4af5f6f93f7c78368e052445fc66e40a034b48420d3747b2994"}
Oct 30 22:00:17.894091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.894050    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" event={"ID":"dd355a55-0491-4cab-97eb-c18e91a5deec","Type":"ContainerStarted","Data":"36414fa29f5e14f185409562997115fe21b6486646101a46ab654b069f57034e"}
Oct 30 22:00:17.895656 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.895633    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"
Oct 30 22:00:17.895805 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.895656    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb" event={"ID":"b44f452c-1628-487d-9edd-2b610efbdea9","Type":"ContainerDied","Data":"52714b262b9e0bf476c0e74c08f49f68ff6b1601b5c01f4b280ce4a7fa9e3746"}
Oct 30 22:00:17.895805 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.895708    5886 scope.go:122] "RemoveContainer" containerID="d5716b03aa3ef212880cca1012f67c498035300a46b2fe487667a427bcf77865"
Oct 30 22:00:17.898939 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.898917    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" event={"ID":"7818ff04-42f0-4d73-90bc-10088989e3a2","Type":"ContainerStarted","Data":"f63bf419cf0812e9097aa8405634a200643634cb62ffbcb71beeac984d09da79"}
Oct 30 22:00:17.981562 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:17.981525    5886 scope.go:122] "RemoveContainer" containerID="8922c95346d27140bcca6c87f23775c023d5695842dc88a17b2cb5643c83d2ff"
Oct 30 22:00:18.058357 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.058327    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"]
Oct 30 22:00:18.064540 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.064492    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9d677d-29889960-z6jvb"]
Oct 30 22:00:18.909673 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.909638    5886 generic.go:356] "Generic (PLEG): container finished" podID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerID="fcf3e9fb2f65d4af5f6f93f7c78368e052445fc66e40a034b48420d3747b2994" exitCode=60
Oct 30 22:00:18.909932 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.909751    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" event={"ID":"fee86e65-5484-4a36-be26-cbc9ccf8c7ab","Type":"ContainerDied","Data":"fcf3e9fb2f65d4af5f6f93f7c78368e052445fc66e40a034b48420d3747b2994"}
Oct 30 22:00:18.909932 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.909808    5886 scope.go:122] "RemoveContainer" containerID="2247bca03dfac1775169845509c6194f9e880eeadc20cd73ad262df5fb5a065b"
Oct 30 22:00:18.910295 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.910275    5886 scope.go:122] "RemoveContainer" containerID="fcf3e9fb2f65d4af5f6f93f7c78368e052445fc66e40a034b48420d3747b2994"
Oct 30 22:00:18.910649 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:18.910626    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq_odh-ai-gateway-infra(fee86e65-5484-4a36-be26-cbc9ccf8c7ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab"
Oct 30 22:00:18.913810 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.913778    5886 generic.go:356] "Generic (PLEG): container finished" podID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerID="36414fa29f5e14f185409562997115fe21b6486646101a46ab654b069f57034e" exitCode=60
Oct 30 22:00:18.913919 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.913830    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" event={"ID":"dd355a55-0491-4cab-97eb-c18e91a5deec","Type":"ContainerDied","Data":"36414fa29f5e14f185409562997115fe21b6486646101a46ab654b069f57034e"}
Oct 30 22:00:18.914347 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.914326    5886 scope.go:122] "RemoveContainer" containerID="36414fa29f5e14f185409562997115fe21b6486646101a46ab654b069f57034e"
Oct 30 22:00:18.914844 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:18.914667    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th_odh-ai-gateway-infra(dd355a55-0491-4cab-97eb-c18e91a5deec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec"
Oct 30 22:00:18.917999 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.917967    5886 generic.go:356] "Generic (PLEG): container finished" podID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerID="f63bf419cf0812e9097aa8405634a200643634cb62ffbcb71beeac984d09da79" exitCode=60
Oct 30 22:00:18.918770 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.918106    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" event={"ID":"7818ff04-42f0-4d73-90bc-10088989e3a2","Type":"ContainerDied","Data":"f63bf419cf0812e9097aa8405634a200643634cb62ffbcb71beeac984d09da79"}
Oct 30 22:00:18.918770 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.918577    5886 scope.go:122] "RemoveContainer" containerID="f63bf419cf0812e9097aa8405634a200643634cb62ffbcb71beeac984d09da79"
Oct 30 22:00:18.919566 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:18.919398    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt_odh-ai-gateway-infra(7818ff04-42f0-4d73-90bc-10088989e3a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2"
Oct 30 22:00:18.920889 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.920867    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" event={"ID":"f720928e-4b4f-438b-b00c-95ee8d3e1141","Type":"ContainerStarted","Data":"38a06e3ca0116d779d0d4fcc1a3d7181b0f63827ebf8ef43eda1c3ee675ccc4b"}
Oct 30 22:00:18.925957 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.925938    5886 scope.go:122] "RemoveContainer" containerID="b1aac802783381f468cf5029d55ddc3dea49d032b04102beeeddaa97b9450b11"
Oct 30 22:00:18.941375 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.941355    5886 scope.go:122] "RemoveContainer" containerID="140477bbba26c1f02a7fe50d0525560ca56fcb03cbfe05fda5ce4966ee74e0a2"
Oct 30 22:00:18.952669 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.952645    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"]
Oct 30 22:00:18.996045 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:18.996010    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"]
Oct 30 22:00:19.034051 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:19.034013    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"]
Oct 30 22:00:19.938311 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:19.938266    5886 generic.go:356] "Generic (PLEG): container finished" podID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerID="38a06e3ca0116d779d0d4fcc1a3d7181b0f63827ebf8ef43eda1c3ee675ccc4b" exitCode=60
Oct 30 22:00:19.938449 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:19.938337    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" event={"ID":"f720928e-4b4f-438b-b00c-95ee8d3e1141","Type":"ContainerDied","Data":"38a06e3ca0116d779d0d4fcc1a3d7181b0f63827ebf8ef43eda1c3ee675ccc4b"}
Oct 30 22:00:19.938449 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:19.938438    5886 scope.go:122] "RemoveContainer" containerID="8922c95346d27140bcca6c87f23775c023d5695842dc88a17b2cb5643c83d2ff"
Oct 30 22:00:19.939058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:19.938943    5886 scope.go:122] "RemoveContainer" containerID="38a06e3ca0116d779d0d4fcc1a3d7181b0f63827ebf8ef43eda1c3ee675ccc4b"
Oct 30 22:00:19.939397 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:19.939368    5886 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t_odh-ai-gateway-infra(f720928e-4b4f-438b-b00c-95ee8d3e1141)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141"
Oct 30 22:00:20.045938 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.045798    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" path="/var/lib/kubelet/pods/b44f452c-1628-487d-9edd-2b610efbdea9/volumes"
Oct 30 22:00:20.079874 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.079847    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"]
Oct 30 22:00:20.217081 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.216968    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:20.235570 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.235546    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:20.255131 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.255105    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:20.390581 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.390550    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd355a55-0491-4cab-97eb-c18e91a5deec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd355a55-0491-4cab-97eb-c18e91a5deec-maas-api-service-ca\") pod \"dd355a55-0491-4cab-97eb-c18e91a5deec\" (UID: \"dd355a55-0491-4cab-97eb-c18e91a5deec\") "
Oct 30 22:00:20.390797 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.390601    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-maas-api-service-ca\") pod \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\" (UID: \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\") "
Oct 30 22:00:20.390797 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.390648    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd355a55-0491-4cab-97eb-c18e91a5deec-kube-api-access-j5sfd\" (UniqueName: \"kubernetes.io/projected/dd355a55-0491-4cab-97eb-c18e91a5deec-kube-api-access-j5sfd\") pod \"dd355a55-0491-4cab-97eb-c18e91a5deec\" (UID: \"dd355a55-0491-4cab-97eb-c18e91a5deec\") "
Oct 30 22:00:20.390797 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.390731    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7818ff04-42f0-4d73-90bc-10088989e3a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7818ff04-42f0-4d73-90bc-10088989e3a2-maas-api-service-ca\") pod \"7818ff04-42f0-4d73-90bc-10088989e3a2\" (UID: \"7818ff04-42f0-4d73-90bc-10088989e3a2\") "
Oct 30 22:00:20.390797 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.390788    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-kube-api-access-7n2rk\" (UniqueName: \"kubernetes.io/projected/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-kube-api-access-7n2rk\") pod \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\" (UID: \"fee86e65-5484-4a36-be26-cbc9ccf8c7ab\") "
Oct 30 22:00:20.391049 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.390806    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7818ff04-42f0-4d73-90bc-10088989e3a2-kube-api-access-w8hrv\" (UniqueName: \"kubernetes.io/projected/7818ff04-42f0-4d73-90bc-10088989e3a2-kube-api-access-w8hrv\") pod \"7818ff04-42f0-4d73-90bc-10088989e3a2\" (UID: \"7818ff04-42f0-4d73-90bc-10088989e3a2\") "
Oct 30 22:00:20.391109 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.390988    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd355a55-0491-4cab-97eb-c18e91a5deec-maas-api-service-ca" pod "dd355a55-0491-4cab-97eb-c18e91a5deec" (UID: "dd355a55-0491-4cab-97eb-c18e91a5deec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.391492 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.391142    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7818ff04-42f0-4d73-90bc-10088989e3a2-maas-api-service-ca" pod "7818ff04-42f0-4d73-90bc-10088989e3a2" (UID: "7818ff04-42f0-4d73-90bc-10088989e3a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.391680 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.391643    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-maas-api-service-ca" pod "fee86e65-5484-4a36-be26-cbc9ccf8c7ab" (UID: "fee86e65-5484-4a36-be26-cbc9ccf8c7ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:20.391840 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.391818    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd355a55-0491-4cab-97eb-c18e91a5deec-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.391924 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.391847    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.391924 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.391862    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7818ff04-42f0-4d73-90bc-10088989e3a2-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.393393 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.393369    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd355a55-0491-4cab-97eb-c18e91a5deec-kube-api-access-j5sfd" pod "dd355a55-0491-4cab-97eb-c18e91a5deec" (UID: "dd355a55-0491-4cab-97eb-c18e91a5deec"). InnerVolumeSpecName "kube-api-access-j5sfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.393468 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.393435    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7818ff04-42f0-4d73-90bc-10088989e3a2-kube-api-access-w8hrv" pod "7818ff04-42f0-4d73-90bc-10088989e3a2" (UID: "7818ff04-42f0-4d73-90bc-10088989e3a2"). InnerVolumeSpecName "kube-api-access-w8hrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.393699 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.393678    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-kube-api-access-7n2rk" pod "fee86e65-5484-4a36-be26-cbc9ccf8c7ab" (UID: "fee86e65-5484-4a36-be26-cbc9ccf8c7ab"). InnerVolumeSpecName "kube-api-access-7n2rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.493213 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.493131    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5sfd\" (UniqueName: \"kubernetes.io/projected/dd355a55-0491-4cab-97eb-c18e91a5deec-kube-api-access-j5sfd\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.493213 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.493211    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n2rk\" (UniqueName: \"kubernetes.io/projected/fee86e65-5484-4a36-be26-cbc9ccf8c7ab-kube-api-access-7n2rk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.493400 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.493223    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8hrv\" (UniqueName: \"kubernetes.io/projected/7818ff04-42f0-4d73-90bc-10088989e3a2-kube-api-access-w8hrv\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:20.950903 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.950872    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"
Oct 30 22:00:20.950903 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.950895    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq" event={"ID":"fee86e65-5484-4a36-be26-cbc9ccf8c7ab","Type":"ContainerDied","Data":"8f5463ab9f8bb33b36c8039ee00f4f0f7d1a62b8180d48558fd17f11c27cbfd3"}
Oct 30 22:00:20.951210 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.950965    5886 scope.go:122] "RemoveContainer" containerID="fcf3e9fb2f65d4af5f6f93f7c78368e052445fc66e40a034b48420d3747b2994"
Oct 30 22:00:20.952674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.952643    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"
Oct 30 22:00:20.952793 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.952703    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th" event={"ID":"dd355a55-0491-4cab-97eb-c18e91a5deec","Type":"ContainerDied","Data":"49d0cfd20f784dfb314f9324fe1d1ed7de90e536e96aae44619bdfa9b737396a"}
Oct 30 22:00:20.954343 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.954320    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"
Oct 30 22:00:20.954472 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.954399    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt" event={"ID":"7818ff04-42f0-4d73-90bc-10088989e3a2","Type":"ContainerDied","Data":"de9d6ccbcc4611f59afa6e4cd8601dc3b34cf60e756143a1922c22b39f1ed831"}
Oct 30 22:00:20.963389 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.963369    5886 scope.go:122] "RemoveContainer" containerID="36414fa29f5e14f185409562997115fe21b6486646101a46ab654b069f57034e"
Oct 30 22:00:20.979233 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:20.979200    5886 scope.go:122] "RemoveContainer" containerID="f63bf419cf0812e9097aa8405634a200643634cb62ffbcb71beeac984d09da79"
Oct 30 22:00:21.062242 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.062138    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"]
Oct 30 22:00:21.067530 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.067505    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b31518-29889960-qx6lq"]
Oct 30 22:00:21.118512 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.118477    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:21.119698 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.119671    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"]
Oct 30 22:00:21.129099 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.129066    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-bfabe1-29889960-zp4zt"]
Oct 30 22:00:21.170721 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.170684    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"]
Oct 30 22:00:21.183253 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.182698    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-64d142-29889960-qk9th"]
Oct 30 22:00:21.222453 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.222349    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f720928e-4b4f-438b-b00c-95ee8d3e1141-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f720928e-4b4f-438b-b00c-95ee8d3e1141-maas-api-service-ca\") pod \"f720928e-4b4f-438b-b00c-95ee8d3e1141\" (UID: \"f720928e-4b4f-438b-b00c-95ee8d3e1141\") "
Oct 30 22:00:21.222647 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.222528    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f720928e-4b4f-438b-b00c-95ee8d3e1141-kube-api-access-tfwjw\" (UniqueName: \"kubernetes.io/projected/f720928e-4b4f-438b-b00c-95ee8d3e1141-kube-api-access-tfwjw\") pod \"f720928e-4b4f-438b-b00c-95ee8d3e1141\" (UID: \"f720928e-4b4f-438b-b00c-95ee8d3e1141\") "
Oct 30 22:00:21.223340 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.223304    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f720928e-4b4f-438b-b00c-95ee8d3e1141-maas-api-service-ca" pod "f720928e-4b4f-438b-b00c-95ee8d3e1141" (UID: "f720928e-4b4f-438b-b00c-95ee8d3e1141"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.224904 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.224877    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f720928e-4b4f-438b-b00c-95ee8d3e1141-maas-api-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.229043 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.228995    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f720928e-4b4f-438b-b00c-95ee8d3e1141-kube-api-access-tfwjw" pod "f720928e-4b4f-438b-b00c-95ee8d3e1141" (UID: "f720928e-4b4f-438b-b00c-95ee8d3e1141"). InnerVolumeSpecName "kube-api-access-tfwjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.325822 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.325781    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfwjw\" (UniqueName: \"kubernetes.io/projected/f720928e-4b4f-438b-b00c-95ee8d3e1141-kube-api-access-tfwjw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:21.966529 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.966498    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"
Oct 30 22:00:21.966529 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.966519    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t" event={"ID":"f720928e-4b4f-438b-b00c-95ee8d3e1141","Type":"ContainerDied","Data":"b9c6bcc13d12eb59a9798e32af3aa7f2bdbef1a594c11549f6538862d4d1dce7"}
Oct 30 22:00:21.966824 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.966606    5886 scope.go:122] "RemoveContainer" containerID="38a06e3ca0116d779d0d4fcc1a3d7181b0f63827ebf8ef43eda1c3ee675ccc4b"
Oct 30 22:00:21.989642 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.989610    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" path="/var/lib/kubelet/pods/7818ff04-42f0-4d73-90bc-10088989e3a2/volumes"
Oct 30 22:00:21.990144 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.990123    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" path="/var/lib/kubelet/pods/dd355a55-0491-4cab-97eb-c18e91a5deec/volumes"
Oct 30 22:00:21.990612 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:21.990591    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" path="/var/lib/kubelet/pods/fee86e65-5484-4a36-be26-cbc9ccf8c7ab/volumes"
Oct 30 22:00:22.015883 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:22.015852    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"]
Oct 30 22:00:22.020839 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:22.020806    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-09eabe-29889960-fhw8t"]
Oct 30 22:00:23.995694 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:23.995658    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" path="/var/lib/kubelet/pods/f720928e-4b4f-438b-b00c-95ee8d3e1141/volumes"
Oct 30 22:00:33.508432 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.508385    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"]
Oct 30 22:00:33.509214 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509193    5886 cpu_manager.go:414] "containerMap: removing container" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:33.509214 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509215    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509227    5886 cpu_manager.go:414] "containerMap: removing container" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509232    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509244    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509250    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509257    5886 cpu_manager.go:414] "containerMap: removing container" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509262    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509273    5886 cpu_manager.go:414] "containerMap: removing container" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509278    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509287    5886 cpu_manager.go:414] "containerMap: removing container" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509293    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509299    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509304    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509313    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509318    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509340    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509345    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509358    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509364    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509371    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:33.509370 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509377    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509395    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509399    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509490    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509499    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509506    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509513    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509518    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509524    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509529    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd355a55-0491-4cab-97eb-c18e91a5deec" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509535    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509543    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509548    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509555    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:33.510355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.509560    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:33.513596 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.513577    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.541566 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.541429    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"openshift-service-ca.crt\""
Oct 30 22:00:33.541566 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.541519    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"routing-overlay\""
Oct 30 22:00:33.542014 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.541993    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"agc-cred-ffd05565871b\""
Oct 30 22:00:33.542014 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.541993    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-plugins-e2e-extmod-b-846656\""
Oct 30 22:00:33.542522 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.542122    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"agc-cred-f060d103b140\""
Oct 30 22:00:33.542628 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.542602    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"payload-processing-external-model-e2e--f95d90a5-dockercfg-5z6mv\""
Oct 30 22:00:33.542814 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.542789    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-846656\"/\"kube-root-ca.crt\""
Oct 30 22:00:33.546246 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.546223    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"]
Oct 30 22:00:33.550354 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.550332    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.550455 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.550363    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-629b4\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-kube-api-access-629b4\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.550637 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.550613    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.550756 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.550727    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.652361 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.652322    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.652592 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.652409    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.652592 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.652543    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.652592 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.652568    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-629b4\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-kube-api-access-629b4\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.653230 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.653202    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.653328 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.653316    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.655200 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.655176    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.671385 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.671357    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-629b4\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-kube-api-access-629b4\") pod \"payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") " pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:33.831202 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:33.831082    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:34.179963 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.179928    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"]
Oct 30 22:00:34.183995 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:00:34.183943    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabd6edf_9d5c_4151_9b48_ac3059989cf6.slice/crio-2084479b7ae79982281ede86c4815a1b78c5accee242c4e691ec3eaa1e0d0126 WatchSource:0}: Error finding container 2084479b7ae79982281ede86c4815a1b78c5accee242c4e691ec3eaa1e0d0126: Status 404 returned error can't find the container with id 2084479b7ae79982281ede86c4815a1b78c5accee242c4e691ec3eaa1e0d0126
Oct 30 22:00:34.466358 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.466264    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"]
Oct 30 22:00:34.467044 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467022    5886 cpu_manager.go:414] "containerMap: removing container" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:34.467103 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467046    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:34.467103 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467062    5886 cpu_manager.go:414] "containerMap: removing container" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:34.467103 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467067    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f720928e-4b4f-438b-b00c-95ee8d3e1141" containerName="cleanup"
Oct 30 22:00:34.467103 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467080    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:34.467103 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467085    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:34.467279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467214    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44f452c-1628-487d-9edd-2b610efbdea9" containerName="cleanup"
Oct 30 22:00:34.467279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467225    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee86e65-5484-4a36-be26-cbc9ccf8c7ab" containerName="cleanup"
Oct 30 22:00:34.467279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.467234    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7818ff04-42f0-4d73-90bc-10088989e3a2" containerName="cleanup"
Oct 30 22:00:34.470964 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.470943    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.485349 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.485319    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"openshift-service-ca.crt\""
Oct 30 22:00:34.485517 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.485351    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"routing-overlay\""
Oct 30 22:00:34.485517 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.485489    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"kube-root-ca.crt\""
Oct 30 22:00:34.490895 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.490865    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"]
Oct 30 22:00:34.496327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.496267    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c19461\""
Oct 30 22:00:34.496327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.496283    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"agc-cred-010bb35e298d\""
Oct 30 22:00:34.496327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.496283    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"agc-cred-8bb6ab2317dd\""
Oct 30 22:00:34.496601 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.496361    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c19461\"/\"payload-processing-external-model-e2e--f2794c3d-dockercfg-pxsz6\""
Oct 30 22:00:34.567108 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.567069    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.567534 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.567279    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.567534 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.567395    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smbft\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-kube-api-access-smbft\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.567534 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.567493    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.668890 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.668855    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.669089 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.669002    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.669194 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.669104    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smbft\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-kube-api-access-smbft\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.669194 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.669184    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.670292 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.670262    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.670428 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.670390    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.672329 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.672305    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.687421 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.687398    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smbft\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-kube-api-access-smbft\") pod \"payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") " pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:34.789260 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:34.789171    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:35.098748 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:35.098709    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"]
Oct 30 22:00:35.100835 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:00:35.100807    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31491d54_82f2_4c85_834e_7de2960df703.slice/crio-cfe8597ac13220fead27497e54c5ebd34e1aa7a6d4a2ae4652ea0daeabd4c08e WatchSource:0}: Error finding container cfe8597ac13220fead27497e54c5ebd34e1aa7a6d4a2ae4652ea0daeabd4c08e: Status 404 returned error can't find the container with id cfe8597ac13220fead27497e54c5ebd34e1aa7a6d4a2ae4652ea0daeabd4c08e
Oct 30 22:00:35.145823 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:35.145786    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx" event={"ID":"31491d54-82f2-4c85-834e-7de2960df703","Type":"ContainerStarted","Data":"cfe8597ac13220fead27497e54c5ebd34e1aa7a6d4a2ae4652ea0daeabd4c08e"}
Oct 30 22:00:35.147575 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:35.147550    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz" event={"ID":"eabd6edf-9d5c-4151-9b48-ac3059989cf6","Type":"ContainerStarted","Data":"1f2d2c5c0d2b8f7029e66c54246a69b4b486dcd35ab217505a1b56244670efb7"}
Oct 30 22:00:35.147716 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:35.147595    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz" event={"ID":"eabd6edf-9d5c-4151-9b48-ac3059989cf6","Type":"ContainerStarted","Data":"2084479b7ae79982281ede86c4815a1b78c5accee242c4e691ec3eaa1e0d0126"}
Oct 30 22:00:35.147716 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:35.147648    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:35.190208 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:35.190133    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz" podStartSLOduration=2.190115018 podStartE2EDuration="2.190115018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:35.184127642 +0000 UTC m=+3035.313379102" watchObservedRunningTime="2026-10-30 22:00:35.190115018 +0000 UTC m=+3035.319366476"
Oct 30 22:00:36.156709 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:36.156668    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx" event={"ID":"31491d54-82f2-4c85-834e-7de2960df703","Type":"ContainerStarted","Data":"e47a765f84d95ae5cfa75a4d4201f5eb12ff9c42d62ab7468342a1e1b2bb8634"}
Oct 30 22:00:36.157123 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:36.156747    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:36.198212 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:36.198134    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx" podStartSLOduration=2.198113465 podStartE2EDuration="2.198113465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:36.191982053 +0000 UTC m=+3036.321233511" watchObservedRunningTime="2026-10-30 22:00:36.198113465 +0000 UTC m=+3036.327365128"
Oct 30 22:00:41.028632 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:41.028596    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:00:46.161117 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:46.161090    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:00:47.170730 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:47.170697    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:00:51.220273 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:51.220236    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"]
Oct 30 22:00:51.220701 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:51.220496    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy" containerID="cri-o://f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182" gracePeriod=30
Oct 30 22:00:56.312766 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.312732    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 22:00:56.313119 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.312794    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 22:00:56.564149 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.564066    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 22:00:56.649216 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649180    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-kube-api-access-bv45g\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-kube-api-access-bv45g\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.649421 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649239    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-socket\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.649421 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649364    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-podinfo\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.649421 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649416    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-envoy\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.649594 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649459    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-credential-socket\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.649594 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649541    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-certs\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.649702 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649619    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-socket" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:56.649702 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649636    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istiod-ca-cert\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.649806 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649762    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-token\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-token\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.649868 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.649826    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-data\") pod \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\" (UID: \"b4109ea5-70d6-484d-b8a2-c78c1a57c889\") "
Oct 30 22:00:56.650039 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.650014    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istiod-ca-cert" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:56.650779 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.650414    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-credential-socket" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:56.650779 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.650610    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:56.650779 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.650632    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:56.650779 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.650646    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:56.650779 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.650725    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-certs" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:56.652925 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.652801    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-data" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:56.653188 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.653128    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-kube-api-access-bv45g" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "kube-api-access-bv45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:56.653681 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.653654    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-podinfo" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:56.654470 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.654444    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-token" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:56.654589 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.654514    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-envoy" pod "b4109ea5-70d6-484d-b8a2-c78c1a57c889" (UID: "b4109ea5-70d6-484d-b8a2-c78c1a57c889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:56.751964 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.751930    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv45g\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-kube-api-access-bv45g\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:56.751964 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.751959    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:56.751964 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.751971    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:56.752218 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.751984    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:56.752218 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.752008    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:56.752218 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:56.752016    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4109ea5-70d6-484d-b8a2-c78c1a57c889-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:00:57.339644 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.339605    5886 generic.go:356] "Generic (PLEG): container finished" podID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerID="f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182" exitCode=0
Oct 30 22:00:57.340090 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.339655    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"
Oct 30 22:00:57.340090 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.339685    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" event={"ID":"b4109ea5-70d6-484d-b8a2-c78c1a57c889","Type":"ContainerDied","Data":"f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182"}
Oct 30 22:00:57.340090 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.339747    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz" event={"ID":"b4109ea5-70d6-484d-b8a2-c78c1a57c889","Type":"ContainerDied","Data":"c3674504a4d634bf94d84465dc3cb14c988d941984dd1469a416a4677faa7b6c"}
Oct 30 22:00:57.340090 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.339771    5886 scope.go:122] "RemoveContainer" containerID="f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182"
Oct 30 22:00:57.353845 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.353602    5886 scope.go:122] "RemoveContainer" containerID="f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182"
Oct 30 22:00:57.354212 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:00:57.354183    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182\": container with ID starting with f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182 not found: ID does not exist" containerID="f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182"
Oct 30 22:00:57.354305 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.354220    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182"} err="failed to get container status \"f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182\": rpc error: code = NotFound desc = could not find container \"f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182\": container with ID starting with f440e060eda17913ae52800cfd4a580f25a33e1c08a22fcc8df16d0d5e971182 not found: ID does not exist"
Oct 30 22:00:57.407330 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.407280    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"]
Oct 30 22:00:57.416340 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.416307    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-jnzlz"]
Oct 30 22:00:57.991342 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:00:57.991304    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" path="/var/lib/kubelet/pods/b4109ea5-70d6-484d-b8a2-c78c1a57c889/volumes"
Oct 30 22:01:11.031689 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:01:11.031586    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:01:33.186658 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.186620    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"]
Oct 30 22:01:33.187678 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.187596    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy"
Oct 30 22:01:33.187678 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.187619    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy"
Oct 30 22:01:33.187882 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.187832    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4109ea5-70d6-484d-b8a2-c78c1a57c889" containerName="istio-proxy"
Oct 30 22:01:33.196171 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.196130    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.210877 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.210852    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"]
Oct 30 22:01:33.216907 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.216883    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ed0866\""
Oct 30 22:01:33.217050 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.216906    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"openshift-service-ca.crt\""
Oct 30 22:01:33.217050 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.216926    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"agc-cred-aec3b6954b2c\""
Oct 30 22:01:33.217050 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.216946    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"agc-cred-ed981606a7c4\""
Oct 30 22:01:33.217050 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.216976    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"routing-overlay\""
Oct 30 22:01:33.221148 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.221110    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"kube-root-ca.crt\""
Oct 30 22:01:33.221148 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.221117    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ed0866\"/\"payload-processing-external-model-e2e--68c3d057-dockercfg-2s4kt\""
Oct 30 22:01:33.348032 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.348001    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.348225 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.348123    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcpx\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-kube-api-access-6wcpx\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.348225 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.348151    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.348320 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.348267    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.449867 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.449769    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.450058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.449873    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcpx\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-kube-api-access-6wcpx\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.450058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.449902    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.450058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.449926    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.450828 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.450798    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.450895 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.450855    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.452437 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.452413    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.469187 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.468371    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcpx\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-kube-api-access-6wcpx\") pod \"payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") " pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.511757 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.511715    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:33.823688 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:33.823665    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"]
Oct 30 22:01:33.826670 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:01:33.826645    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2e229b_c0f6_4b75_9fc8_8f900efea735.slice/crio-0340560e8382b6d9e7e20baca2acf0a18ec22bfc6bd0be0c0443d83dd8206913 WatchSource:0}: Error finding container 0340560e8382b6d9e7e20baca2acf0a18ec22bfc6bd0be0c0443d83dd8206913: Status 404 returned error can't find the container with id 0340560e8382b6d9e7e20baca2acf0a18ec22bfc6bd0be0c0443d83dd8206913
Oct 30 22:01:34.721970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:34.721921    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7" event={"ID":"1a2e229b-c0f6-4b75-9fc8-8f900efea735","Type":"ContainerStarted","Data":"b0aab307d09860459125102db6a10a6a8e523fa63bce9adad7677312e913bb80"}
Oct 30 22:01:34.722425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:34.721989    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7" event={"ID":"1a2e229b-c0f6-4b75-9fc8-8f900efea735","Type":"ContainerStarted","Data":"0340560e8382b6d9e7e20baca2acf0a18ec22bfc6bd0be0c0443d83dd8206913"}
Oct 30 22:01:34.722425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:34.722020    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:01:34.759009 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:34.758952    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7" podStartSLOduration=1.75893882 podStartE2EDuration="1.75893882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:34.758207563 +0000 UTC m=+3094.887459021" watchObservedRunningTime="2026-10-30 22:01:34.75893882 +0000 UTC m=+3094.888190331"
Oct 30 22:01:41.030214 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:01:41.030175    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:01:45.733198 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:01:45.733134    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:02:11.030833 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:02:11.030794    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:02:41.030944 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:02:41.030850    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:03:11.029145 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:11.029105    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:03:32.731910 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:32.731874    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"]
Oct 30 22:03:34.771661 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:34.771625    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"]
Oct 30 22:03:37.733051 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:37.733009    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx" podUID="31491d54-82f2-4c85-834e-7de2960df703" containerName="payload-processing" containerID="cri-o://e47a765f84d95ae5cfa75a4d4201f5eb12ff9c42d62ab7468342a1e1b2bb8634" gracePeriod=25
Oct 30 22:03:37.958994 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:37.958958    5886 generic.go:356] "Generic (PLEG): container finished" podID="31491d54-82f2-4c85-834e-7de2960df703" containerID="e47a765f84d95ae5cfa75a4d4201f5eb12ff9c42d62ab7468342a1e1b2bb8634" exitCode=0
Oct 30 22:03:37.959232 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:37.959211    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx" event={"ID":"31491d54-82f2-4c85-834e-7de2960df703","Type":"ContainerDied","Data":"e47a765f84d95ae5cfa75a4d4201f5eb12ff9c42d62ab7468342a1e1b2bb8634"}
Oct 30 22:03:38.023695 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.023661    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:03:38.119345 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.119312    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-plugins-config-volume\") pod \"31491d54-82f2-4c85-834e-7de2960df703\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") "
Oct 30 22:03:38.119530 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.119371    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-kube-api-access-smbft\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-kube-api-access-smbft\") pod \"31491d54-82f2-4c85-834e-7de2960df703\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") "
Oct 30 22:03:38.119625 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.119551    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-provider-credentials\") pod \"31491d54-82f2-4c85-834e-7de2960df703\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") "
Oct 30 22:03:38.119695 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.119680    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-routing-overlay-volume\") pod \"31491d54-82f2-4c85-834e-7de2960df703\" (UID: \"31491d54-82f2-4c85-834e-7de2960df703\") "
Oct 30 22:03:38.119756 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.119733    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-plugins-config-volume" pod "31491d54-82f2-4c85-834e-7de2960df703" (UID: "31491d54-82f2-4c85-834e-7de2960df703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:38.120031 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.120003    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-routing-overlay-volume" pod "31491d54-82f2-4c85-834e-7de2960df703" (UID: "31491d54-82f2-4c85-834e-7de2960df703"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:38.120977 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.120948    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:38.121092 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.120982    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31491d54-82f2-4c85-834e-7de2960df703-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:38.121972 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.121941    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-provider-credentials" pod "31491d54-82f2-4c85-834e-7de2960df703" (UID: "31491d54-82f2-4c85-834e-7de2960df703"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:38.122581 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.122554    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-kube-api-access-smbft" pod "31491d54-82f2-4c85-834e-7de2960df703" (UID: "31491d54-82f2-4c85-834e-7de2960df703"). InnerVolumeSpecName "kube-api-access-smbft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:38.222415 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.222377    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smbft\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-kube-api-access-smbft\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:38.222415 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.222407    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31491d54-82f2-4c85-834e-7de2960df703-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:38.969654 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.969621    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"
Oct 30 22:03:38.969654 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.969644    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx" event={"ID":"31491d54-82f2-4c85-834e-7de2960df703","Type":"ContainerDied","Data":"cfe8597ac13220fead27497e54c5ebd34e1aa7a6d4a2ae4652ea0daeabd4c08e"}
Oct 30 22:03:38.970168 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:38.969706    5886 scope.go:122] "RemoveContainer" containerID="e47a765f84d95ae5cfa75a4d4201f5eb12ff9c42d62ab7468342a1e1b2bb8634"
Oct 30 22:03:39.018433 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:39.018395    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"]
Oct 30 22:03:39.030097 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:39.030068    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c19461/payload-processing-external-model-e2e-extmod-a-c19461-548fbfszx"]
Oct 30 22:03:39.772837 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:39.772791    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz" podUID="eabd6edf-9d5c-4151-9b48-ac3059989cf6" containerName="payload-processing" containerID="cri-o://1f2d2c5c0d2b8f7029e66c54246a69b4b486dcd35ab217505a1b56244670efb7" gracePeriod=25
Oct 30 22:03:40.004519 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.004454    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31491d54-82f2-4c85-834e-7de2960df703" path="/var/lib/kubelet/pods/31491d54-82f2-4c85-834e-7de2960df703/volumes"
Oct 30 22:03:40.010909 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.010870    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz" event={"ID":"eabd6edf-9d5c-4151-9b48-ac3059989cf6","Type":"ContainerDied","Data":"1f2d2c5c0d2b8f7029e66c54246a69b4b486dcd35ab217505a1b56244670efb7"}
Oct 30 22:03:40.011084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.009946    5886 generic.go:356] "Generic (PLEG): container finished" podID="eabd6edf-9d5c-4151-9b48-ac3059989cf6" containerID="1f2d2c5c0d2b8f7029e66c54246a69b4b486dcd35ab217505a1b56244670efb7" exitCode=0
Oct 30 22:03:40.115662 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.115631    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:03:40.283734 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.283606    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-provider-credentials\") pod \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") "
Oct 30 22:03:40.283734 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.283683    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-routing-overlay-volume\") pod \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") "
Oct 30 22:03:40.283992 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.283801    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-plugins-config-volume\") pod \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") "
Oct 30 22:03:40.283992 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.283845    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-kube-api-access-629b4\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-kube-api-access-629b4\") pod \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\" (UID: \"eabd6edf-9d5c-4151-9b48-ac3059989cf6\") "
Oct 30 22:03:40.290137 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.290079    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-plugins-config-volume" pod "eabd6edf-9d5c-4151-9b48-ac3059989cf6" (UID: "eabd6edf-9d5c-4151-9b48-ac3059989cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:40.291124 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.291096    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-routing-overlay-volume" pod "eabd6edf-9d5c-4151-9b48-ac3059989cf6" (UID: "eabd6edf-9d5c-4151-9b48-ac3059989cf6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:40.347222 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.347087    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-kube-api-access-629b4" pod "eabd6edf-9d5c-4151-9b48-ac3059989cf6" (UID: "eabd6edf-9d5c-4151-9b48-ac3059989cf6"). InnerVolumeSpecName "kube-api-access-629b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:40.377922 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.365567    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-provider-credentials" pod "eabd6edf-9d5c-4151-9b48-ac3059989cf6" (UID: "eabd6edf-9d5c-4151-9b48-ac3059989cf6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:40.396454 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.396414    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:40.396689 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.396666    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:40.396810 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.396797    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eabd6edf-9d5c-4151-9b48-ac3059989cf6-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:40.396900 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:40.396890    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-629b4\" (UniqueName: \"kubernetes.io/projected/eabd6edf-9d5c-4151-9b48-ac3059989cf6-kube-api-access-629b4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:41.021225 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.021189    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz" event={"ID":"eabd6edf-9d5c-4151-9b48-ac3059989cf6","Type":"ContainerDied","Data":"2084479b7ae79982281ede86c4815a1b78c5accee242c4e691ec3eaa1e0d0126"}
Oct 30 22:03:41.021225 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.021222    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"
Oct 30 22:03:41.021740 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.021259    5886 scope.go:122] "RemoveContainer" containerID="1f2d2c5c0d2b8f7029e66c54246a69b4b486dcd35ab217505a1b56244670efb7"
Oct 30 22:03:41.035203 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:41.035146    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:03:41.098396 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.098366    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"]
Oct 30 22:03:41.110281 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.110246    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-846656/payload-processing-external-model-e2e-extmod-b-846656-6fd6zvbfz"]
Oct 30 22:03:41.724036 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.724003    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"]
Oct 30 22:03:41.724677 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.724663    5886 cpu_manager.go:414] "containerMap: removing container" podUID="eabd6edf-9d5c-4151-9b48-ac3059989cf6" containerName="payload-processing"
Oct 30 22:03:41.724745 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.724679    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabd6edf-9d5c-4151-9b48-ac3059989cf6" containerName="payload-processing"
Oct 30 22:03:41.724745 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.724712    5886 cpu_manager.go:414] "containerMap: removing container" podUID="31491d54-82f2-4c85-834e-7de2960df703" containerName="payload-processing"
Oct 30 22:03:41.724745 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.724718    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31491d54-82f2-4c85-834e-7de2960df703" containerName="payload-processing"
Oct 30 22:03:41.724837 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.724818    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="31491d54-82f2-4c85-834e-7de2960df703" containerName="payload-processing"
Oct 30 22:03:41.724837 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.724833    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabd6edf-9d5c-4151-9b48-ac3059989cf6" containerName="payload-processing"
Oct 30 22:03:41.729999 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.729977    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.747151 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.747124    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 22:03:41.747151 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.747174    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"]
Oct 30 22:03:41.754658 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.754611    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 22:03:41.754658 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.754649    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bba74b814b2d\""
Oct 30 22:03:41.754870 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.754651    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d9c68187ac0c\""
Oct 30 22:03:41.754870 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.754698    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-73353bd0e503\""
Oct 30 22:03:41.758585 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.758550    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rxxkz\""
Oct 30 22:03:41.816450 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.816419    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 22:03:41.816450 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.816420    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 22:03:41.831862 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.831829    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5mm\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-kube-api-access-nq5mm\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.832026 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.831912    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-routing-overlay-volume\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.832143 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.832122    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-provider-credentials\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.832238 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.832197    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-plugins-config-volume\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.938049 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.938011    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5mm\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-kube-api-access-nq5mm\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.938278 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.938069    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-routing-overlay-volume\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.938278 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.938246    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-provider-credentials\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.938421 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.938297    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-plugins-config-volume\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.939088 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.939062    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-routing-overlay-volume\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.939361 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.939338    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-plugins-config-volume\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.946195 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.944829    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-provider-credentials\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.955837 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.955813    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5mm\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-kube-api-access-nq5mm\") pod \"payload-processing-external-model-5ff676d685-np2rw\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") " pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:41.990599 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:41.990517    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabd6edf-9d5c-4151-9b48-ac3059989cf6" path="/var/lib/kubelet/pods/eabd6edf-9d5c-4151-9b48-ac3059989cf6/volumes"
Oct 30 22:03:42.044773 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:42.044740    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:42.362003 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:42.361917    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"]
Oct 30 22:03:42.366635 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:03:42.366587    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0aee4e_724f_4628_b3be_038109283c71.slice/crio-b748fa5916c69994f4a2f45f8cf27929583a268013acd030123f4147716b4fcb WatchSource:0}: Error finding container b748fa5916c69994f4a2f45f8cf27929583a268013acd030123f4147716b4fcb: Status 404 returned error can't find the container with id b748fa5916c69994f4a2f45f8cf27929583a268013acd030123f4147716b4fcb
Oct 30 22:03:43.053096 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:43.053042    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw" event={"ID":"ac0aee4e-724f-4628-b3be-038109283c71","Type":"ContainerStarted","Data":"0845de9f8420c0a6a8494b6a47e226dec19b406b24a3e2a32548e1bfb8734e6d"}
Oct 30 22:03:43.053579 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:43.053142    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw" event={"ID":"ac0aee4e-724f-4628-b3be-038109283c71","Type":"ContainerStarted","Data":"b748fa5916c69994f4a2f45f8cf27929583a268013acd030123f4147716b4fcb"}
Oct 30 22:03:43.053579 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:43.053209    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:43.091117 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:43.091059    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw" podStartSLOduration=2.09104558 podStartE2EDuration="2.09104558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:43.088867099 +0000 UTC m=+3223.218118557" watchObservedRunningTime="2026-10-30 22:03:43.09104558 +0000 UTC m=+3223.220297058"
Oct 30 22:03:47.316260 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.312192    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"]
Oct 30 22:03:47.329361 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.329331    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:47.346948 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.346909    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"]
Oct 30 22:03:47.385548 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.385485    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.465179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.462822    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.465179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.462898    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.465179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.462937    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.465179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.463046    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72x7\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-kube-api-access-t72x7\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.488178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.478280    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"]
Oct 30 22:03:47.620203 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.603728    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.620203 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.603803    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.620203 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.603836    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.620203 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.603945    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72x7\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-kube-api-access-t72x7\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.620203 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.618969    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-plugins-config-volume\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.620203 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.620037    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-routing-overlay-volume\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.624252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.624198    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72x7\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-kube-api-access-t72x7\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.684178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.683224    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials\") pod \"payload-processing-external-model-6cb6ff46b5-254dl\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") " pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:47.754176 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:47.753454    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:48.947118 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:48.947072    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"]
Oct 30 22:03:49.634257 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:49.633854    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl" event={"ID":"24ce7e6f-cb86-466f-8469-e32a9074067f","Type":"ContainerStarted","Data":"26161fd451424ad314240fc862fc097191932bebb45caf493f3e6642463a72d7"}
Oct 30 22:03:49.634257 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:49.633920    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl" event={"ID":"24ce7e6f-cb86-466f-8469-e32a9074067f","Type":"ContainerStarted","Data":"21655f0dfa44f011ad6d8838d81ab48fa9b84d14b8dbdedda39291227711e55c"}
Oct 30 22:03:49.634662 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:49.634639    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:49.656768 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:49.656733    5886 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bba74b814b2d: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:49.656768 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:49.656773    5886 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:49.657021 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:49.656856    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials podName:24ce7e6f-cb86-466f-8469-e32a9074067f nodeName:}" failed. No retries permitted until 2026-10-30 22:03:50.156833684 +0000 UTC m=+3230.286085122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials") pod "payload-processing-external-model-6cb6ff46b5-254dl" (UID: "24ce7e6f-cb86-466f-8469-e32a9074067f") : secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:49.679586 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:49.679515    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl" podStartSLOduration=2.679497001 podStartE2EDuration="2.679497001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:49.674082933 +0000 UTC m=+3229.803334391" watchObservedRunningTime="2026-10-30 22:03:49.679497001 +0000 UTC m=+3229.808748461"
Oct 30 22:03:50.168901 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:50.165730    5886 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bba74b814b2d: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:50.168901 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:50.165775    5886 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:50.168901 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:50.165842    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials podName:24ce7e6f-cb86-466f-8469-e32a9074067f nodeName:}" failed. No retries permitted until 2026-10-30 22:03:51.165821626 +0000 UTC m=+3231.295073064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials") pod "payload-processing-external-model-6cb6ff46b5-254dl" (UID: "24ce7e6f-cb86-466f-8469-e32a9074067f") : secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:50.214349 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:50.214306    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"]
Oct 30 22:03:51.181187 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:51.180870    5886 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bba74b814b2d: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:51.181187 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:51.180915    5886 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:51.181187 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:51.180978    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials podName:24ce7e6f-cb86-466f-8469-e32a9074067f nodeName:}" failed. No retries permitted until 2026-10-30 22:03:53.180959257 +0000 UTC m=+3233.310210692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials") pod "payload-processing-external-model-6cb6ff46b5-254dl" (UID: "24ce7e6f-cb86-466f-8469-e32a9074067f") : secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:52.313305 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.313229    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw" podUID="ac0aee4e-724f-4628-b3be-038109283c71" containerName="payload-processing" containerID="cri-o://0845de9f8420c0a6a8494b6a47e226dec19b406b24a3e2a32548e1bfb8734e6d" gracePeriod=25
Oct 30 22:03:52.830988 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.830949    5886 generic.go:356] "Generic (PLEG): container finished" podID="ac0aee4e-724f-4628-b3be-038109283c71" containerID="0845de9f8420c0a6a8494b6a47e226dec19b406b24a3e2a32548e1bfb8734e6d" exitCode=0
Oct 30 22:03:52.831289 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.831053    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw" event={"ID":"ac0aee4e-724f-4628-b3be-038109283c71","Type":"ContainerDied","Data":"0845de9f8420c0a6a8494b6a47e226dec19b406b24a3e2a32548e1bfb8734e6d"}
Oct 30 22:03:52.843699 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.833584    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:52.964261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.964223    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-routing-overlay-volume\") pod \"ac0aee4e-724f-4628-b3be-038109283c71\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") "
Oct 30 22:03:52.964459 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.964415    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-plugins-config-volume\") pod \"ac0aee4e-724f-4628-b3be-038109283c71\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") "
Oct 30 22:03:52.964532 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.964469    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-provider-credentials\") pod \"ac0aee4e-724f-4628-b3be-038109283c71\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") "
Oct 30 22:03:52.964587 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.964540    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-kube-api-access-nq5mm\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-kube-api-access-nq5mm\") pod \"ac0aee4e-724f-4628-b3be-038109283c71\" (UID: \"ac0aee4e-724f-4628-b3be-038109283c71\") "
Oct 30 22:03:52.964687 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.964652    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-routing-overlay-volume" pod "ac0aee4e-724f-4628-b3be-038109283c71" (UID: "ac0aee4e-724f-4628-b3be-038109283c71"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:52.965021 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.964996    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-plugins-config-volume" pod "ac0aee4e-724f-4628-b3be-038109283c71" (UID: "ac0aee4e-724f-4628-b3be-038109283c71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:52.965261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.965242    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:52.965345 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.965266    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac0aee4e-724f-4628-b3be-038109283c71-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:52.967433 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.967400    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-kube-api-access-nq5mm" pod "ac0aee4e-724f-4628-b3be-038109283c71" (UID: "ac0aee4e-724f-4628-b3be-038109283c71"). InnerVolumeSpecName "kube-api-access-nq5mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:52.968291 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:52.968268    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-provider-credentials" pod "ac0aee4e-724f-4628-b3be-038109283c71" (UID: "ac0aee4e-724f-4628-b3be-038109283c71"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:53.066761 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.066716    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:53.066761 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.066746    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq5mm\" (UniqueName: \"kubernetes.io/projected/ac0aee4e-724f-4628-b3be-038109283c71-kube-api-access-nq5mm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:53.287238 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:53.273498    5886 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bba74b814b2d: secret "agc-cred-bba74b814b2d" not found
Oct 30 22:03:53.287560 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:53.287531    5886 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d9c68187ac0c: secret "agc-cred-d9c68187ac0c" not found
Oct 30 22:03:53.287684 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:53.287670    5886 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl: [secret "agc-cred-bba74b814b2d" not found, secret "agc-cred-d9c68187ac0c" not found]
Oct 30 22:03:53.287860 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:03:53.287847    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials podName:24ce7e6f-cb86-466f-8469-e32a9074067f nodeName:}" failed. No retries permitted until 2026-10-30 22:03:57.28782332 +0000 UTC m=+3237.417074757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials") pod "payload-processing-external-model-6cb6ff46b5-254dl" (UID: "24ce7e6f-cb86-466f-8469-e32a9074067f") : [secret "agc-cred-bba74b814b2d" not found, secret "agc-cred-d9c68187ac0c" not found]
Oct 30 22:03:53.578466 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.578385    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"]
Oct 30 22:03:53.581101 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.581067    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ac0aee4e-724f-4628-b3be-038109283c71" containerName="payload-processing"
Oct 30 22:03:53.581101 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.581095    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0aee4e-724f-4628-b3be-038109283c71" containerName="payload-processing"
Oct 30 22:03:53.581341 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.581276    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0aee4e-724f-4628-b3be-038109283c71" containerName="payload-processing"
Oct 30 22:03:53.585480 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.585460    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.609955 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.609925    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"]
Oct 30 22:03:53.613114 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.613008    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"kube-root-ca.crt\""
Oct 30 22:03:53.613114 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.613054    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"agc-cred-284266574f4c\""
Oct 30 22:03:53.613114 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.613075    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"openshift-service-ca.crt\""
Oct 30 22:03:53.613357 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.613128    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"routing-overlay\""
Oct 30 22:03:53.613357 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.613205    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3d5d13\""
Oct 30 22:03:53.617168 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.617140    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3d5d13\"/\"payload-processing-external-model-e2e--4472330f-dockercfg-tht88\""
Oct 30 22:03:53.681788 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.681746    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.681988 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.681805    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.682063 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.682006    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.682122 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.682068    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9q6j\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-kube-api-access-h9q6j\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.785494 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.785454    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.787864 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.787834    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.788121 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.788104    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9q6j\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-kube-api-access-h9q6j\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.788410 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.788377    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.789671 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.789648    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.790515 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.790484    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.817536 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.817399    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.853300 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.839867    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9q6j\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-kube-api-access-h9q6j\") pod \"payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") " pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.861939 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.861900    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw" event={"ID":"ac0aee4e-724f-4628-b3be-038109283c71","Type":"ContainerDied","Data":"b748fa5916c69994f4a2f45f8cf27929583a268013acd030123f4147716b4fcb"}
Oct 30 22:03:53.862253 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.862234    5886 scope.go:122] "RemoveContainer" containerID="0845de9f8420c0a6a8494b6a47e226dec19b406b24a3e2a32548e1bfb8734e6d"
Oct 30 22:03:53.862567 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.862548    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"
Oct 30 22:03:53.904089 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.904018    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:53.939202 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.937367    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"]
Oct 30 22:03:53.941108 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:53.941077    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5ff676d685-np2rw"]
Oct 30 22:03:54.002637 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:54.002604    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0aee4e-724f-4628-b3be-038109283c71" path="/var/lib/kubelet/pods/ac0aee4e-724f-4628-b3be-038109283c71/volumes"
Oct 30 22:03:54.226833 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:54.226809    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"]
Oct 30 22:03:54.229289 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:03:54.229260    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662816b4_6b02_436a_92b8_000cd7ae75c0.slice/crio-464bcfef91f2d93df84cad42beca0b812e892aeacc329e7a65fc6eda3217ee18 WatchSource:0}: Error finding container 464bcfef91f2d93df84cad42beca0b812e892aeacc329e7a65fc6eda3217ee18: Status 404 returned error can't find the container with id 464bcfef91f2d93df84cad42beca0b812e892aeacc329e7a65fc6eda3217ee18
Oct 30 22:03:54.872824 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:54.872729    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj" event={"ID":"662816b4-6b02-436a-92b8-000cd7ae75c0","Type":"ContainerStarted","Data":"a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90"}
Oct 30 22:03:54.872824 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:54.872797    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj" event={"ID":"662816b4-6b02-436a-92b8-000cd7ae75c0","Type":"ContainerStarted","Data":"464bcfef91f2d93df84cad42beca0b812e892aeacc329e7a65fc6eda3217ee18"}
Oct 30 22:03:54.873295 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:54.872848    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:03:54.911097 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:54.911043    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj" podStartSLOduration=1.911028092 podStartE2EDuration="1.911028092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:54.90998789 +0000 UTC m=+3235.039239357" watchObservedRunningTime="2026-10-30 22:03:54.911028092 +0000 UTC m=+3235.040279549"
Oct 30 22:03:56.649976 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:56.649916    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl" podUID="24ce7e6f-cb86-466f-8469-e32a9074067f" containerName="payload-processing" containerID="cri-o://26161fd451424ad314240fc862fc097191932bebb45caf493f3e6642463a72d7" gracePeriod=25
Oct 30 22:03:57.014524 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.014043    5886 generic.go:356] "Generic (PLEG): container finished" podID="24ce7e6f-cb86-466f-8469-e32a9074067f" containerID="26161fd451424ad314240fc862fc097191932bebb45caf493f3e6642463a72d7" exitCode=0
Oct 30 22:03:57.014524 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.014382    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl" event={"ID":"24ce7e6f-cb86-466f-8469-e32a9074067f","Type":"ContainerDied","Data":"26161fd451424ad314240fc862fc097191932bebb45caf493f3e6642463a72d7"}
Oct 30 22:03:57.099578 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.099537    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:57.163555 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.163518    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-plugins-config-volume\") pod \"24ce7e6f-cb86-466f-8469-e32a9074067f\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") "
Oct 30 22:03:57.163748 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.163628    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-routing-overlay-volume\") pod \"24ce7e6f-cb86-466f-8469-e32a9074067f\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") "
Oct 30 22:03:57.163748 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.163664    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-kube-api-access-t72x7\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-kube-api-access-t72x7\") pod \"24ce7e6f-cb86-466f-8469-e32a9074067f\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") "
Oct 30 22:03:57.163869 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.163842    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials\") pod \"24ce7e6f-cb86-466f-8469-e32a9074067f\" (UID: \"24ce7e6f-cb86-466f-8469-e32a9074067f\") "
Oct 30 22:03:57.164009 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.163976    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-plugins-config-volume" pod "24ce7e6f-cb86-466f-8469-e32a9074067f" (UID: "24ce7e6f-cb86-466f-8469-e32a9074067f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:57.164396 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.164371    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-routing-overlay-volume" pod "24ce7e6f-cb86-466f-8469-e32a9074067f" (UID: "24ce7e6f-cb86-466f-8469-e32a9074067f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:57.164665 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.164645    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:57.164891 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.164871    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24ce7e6f-cb86-466f-8469-e32a9074067f-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:57.166883 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.166858    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials" pod "24ce7e6f-cb86-466f-8469-e32a9074067f" (UID: "24ce7e6f-cb86-466f-8469-e32a9074067f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:57.166993 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.166894    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-kube-api-access-t72x7" pod "24ce7e6f-cb86-466f-8469-e32a9074067f" (UID: "24ce7e6f-cb86-466f-8469-e32a9074067f"). InnerVolumeSpecName "kube-api-access-t72x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:57.266828 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.266730    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:57.266828 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.266769    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t72x7\" (UniqueName: \"kubernetes.io/projected/24ce7e6f-cb86-466f-8469-e32a9074067f-kube-api-access-t72x7\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:03:57.862368 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.862336    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"]
Oct 30 22:03:57.863079 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.863047    5886 cpu_manager.go:414] "containerMap: removing container" podUID="24ce7e6f-cb86-466f-8469-e32a9074067f" containerName="payload-processing"
Oct 30 22:03:57.863079 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.863067    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ce7e6f-cb86-466f-8469-e32a9074067f" containerName="payload-processing"
Oct 30 22:03:57.863304 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.863292    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ce7e6f-cb86-466f-8469-e32a9074067f" containerName="payload-processing"
Oct 30 22:03:57.867507 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.867477    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:57.896347 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.896317    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"agc-cred-899505401fa4\""
Oct 30 22:03:57.896347 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.896349    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-plugins-e2e-extmod-b-db1288\""
Oct 30 22:03:57.896613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.896429    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"payload-processing-external-model-e2e--f0c0883f-dockercfg-xxqq7\""
Oct 30 22:03:57.896613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.896428    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"openshift-service-ca.crt\""
Oct 30 22:03:57.896613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.896446    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"kube-root-ca.crt\""
Oct 30 22:03:57.896613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.896510    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-db1288\"/\"routing-overlay\""
Oct 30 22:03:57.901828 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.901799    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"]
Oct 30 22:03:57.995276 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.995235    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:57.995487 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.995404    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:57.995624 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.995485    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:57.995727 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:57.995653    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5278d\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-kube-api-access-5278d\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.081380 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.071686    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl" event={"ID":"24ce7e6f-cb86-466f-8469-e32a9074067f","Type":"ContainerDied","Data":"21655f0dfa44f011ad6d8838d81ab48fa9b84d14b8dbdedda39291227711e55c"}
Oct 30 22:03:58.081380 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.071766    5886 scope.go:122] "RemoveContainer" containerID="26161fd451424ad314240fc862fc097191932bebb45caf493f3e6642463a72d7"
Oct 30 22:03:58.081380 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.073253    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"
Oct 30 22:03:58.113261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.105590    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.113261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.105714    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.113261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.105813    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.113261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.106031    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5278d\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-kube-api-access-5278d\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.113261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.107704    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.113261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.109716    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.142064 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.141953    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"]
Oct 30 22:03:58.151506 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.151459    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cb6ff46b5-254dl"]
Oct 30 22:03:58.155227 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.155196    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.159721 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.159308    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5278d\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-kube-api-access-5278d\") pod \"payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") " pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.184071 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.184034    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:58.489378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.489337    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"]
Oct 30 22:03:58.493802 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:03:58.493768    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c8c5dc_b820_4d48_84c9_8f65e292689a.slice/crio-18be67fda1e2c8273cd0691eb0b33f1e1dc872333d019481debb54e0054eeb2d WatchSource:0}: Error finding container 18be67fda1e2c8273cd0691eb0b33f1e1dc872333d019481debb54e0054eeb2d: Status 404 returned error can't find the container with id 18be67fda1e2c8273cd0691eb0b33f1e1dc872333d019481debb54e0054eeb2d
Oct 30 22:03:58.814268 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.814183    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"]
Oct 30 22:03:58.819568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.819543    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:03:58.834674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.833520    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a42c31-dockercfg-7c6pc\""
Oct 30 22:03:58.834674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.833575    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a42c31\""
Oct 30 22:03:58.834674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.834339    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"]
Oct 30 22:03:58.919940 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.919592    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a39adb4-8217-40e0-9e5a-843e148a49c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk\" (UID: \"5a39adb4-8217-40e0-9e5a-843e148a49c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:03:58.919940 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:58.919780    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9j5\" (UniqueName: \"kubernetes.io/projected/5a39adb4-8217-40e0-9e5a-843e148a49c5-kube-api-access-9v9j5\") pod \"payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk\" (UID: \"5a39adb4-8217-40e0-9e5a-843e148a49c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:03:59.037238 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.037201    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a39adb4-8217-40e0-9e5a-843e148a49c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk\" (UID: \"5a39adb4-8217-40e0-9e5a-843e148a49c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:03:59.037607 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.037583    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9j5\" (UniqueName: \"kubernetes.io/projected/5a39adb4-8217-40e0-9e5a-843e148a49c5-kube-api-access-9v9j5\") pod \"payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk\" (UID: \"5a39adb4-8217-40e0-9e5a-843e148a49c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:03:59.038980 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.038956    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a39adb4-8217-40e0-9e5a-843e148a49c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk\" (UID: \"5a39adb4-8217-40e0-9e5a-843e148a49c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:03:59.122232 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.119079    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9j5\" (UniqueName: \"kubernetes.io/projected/5a39adb4-8217-40e0-9e5a-843e148a49c5-kube-api-access-9v9j5\") pod \"payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk\" (UID: \"5a39adb4-8217-40e0-9e5a-843e148a49c5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:03:59.129650 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.129427    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz" event={"ID":"e7c8c5dc-b820-4d48-84c9-8f65e292689a","Type":"ContainerStarted","Data":"d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d"}
Oct 30 22:03:59.129650 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.129494    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz" event={"ID":"e7c8c5dc-b820-4d48-84c9-8f65e292689a","Type":"ContainerStarted","Data":"18be67fda1e2c8273cd0691eb0b33f1e1dc872333d019481debb54e0054eeb2d"}
Oct 30 22:03:59.131240 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.131180    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:03:59.135558 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.135530    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:03:59.450796 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.450746    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz" podStartSLOduration=2.450733441 podStartE2EDuration="2.450733441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:59.170177143 +0000 UTC m=+3239.299428594" watchObservedRunningTime="2026-10-30 22:03:59.450733441 +0000 UTC m=+3239.579984956"
Oct 30 22:03:59.453244 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.453198    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"]
Oct 30 22:03:59.456221 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:03:59.456192    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a39adb4_8217_40e0_9e5a_843e148a49c5.slice/crio-bc7c54dba4f2f609771c8732a3f7fa3b23ff03098324b4ebf3267f14078fdcf4 WatchSource:0}: Error finding container bc7c54dba4f2f609771c8732a3f7fa3b23ff03098324b4ebf3267f14078fdcf4: Status 404 returned error can't find the container with id bc7c54dba4f2f609771c8732a3f7fa3b23ff03098324b4ebf3267f14078fdcf4
Oct 30 22:03:59.992884 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:03:59.992842    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ce7e6f-cb86-466f-8469-e32a9074067f" path="/var/lib/kubelet/pods/24ce7e6f-cb86-466f-8469-e32a9074067f/volumes"
Oct 30 22:04:00.145957 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:00.145903    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk" event={"ID":"5a39adb4-8217-40e0-9e5a-843e148a49c5","Type":"ContainerStarted","Data":"9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90"}
Oct 30 22:04:00.146134 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:00.146004    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk" event={"ID":"5a39adb4-8217-40e0-9e5a-843e148a49c5","Type":"ContainerStarted","Data":"bc7c54dba4f2f609771c8732a3f7fa3b23ff03098324b4ebf3267f14078fdcf4"}
Oct 30 22:04:00.146349 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:00.146335    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:04:00.184102 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:00.184039    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk" podStartSLOduration=2.184021332 podStartE2EDuration="2.184021332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:00.181603317 +0000 UTC m=+3240.310854776" watchObservedRunningTime="2026-10-30 22:04:00.184021332 +0000 UTC m=+3240.313272791"
Oct 30 22:04:05.247446 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:05.247410    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"]
Oct 30 22:04:05.250572 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:05.250546    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:04:05.917731 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:05.917701    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:04:09.023921 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.023882    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"]
Oct 30 22:04:09.030281 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.030255    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.050980 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.050949    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"]
Oct 30 22:04:09.055327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.055303    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-846656-openshift-default-dockercfg-sl8ph\""
Oct 30 22:04:09.089957 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.089924    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.090150 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.089973    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.090150 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.090068    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.090276 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.090249    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.090390 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.090369    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6054e7ae-175c-460b-8de7-6c27cbf501a3-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.090447 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.090430    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6gj\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-kube-api-access-sf6gj\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.090553 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.090534    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.090701 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.090681    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.090782 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.090724    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192014 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.191974    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192014 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192020    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192268 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192084    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192268 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192116    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192268 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192146    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192268 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192244    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192502 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192317    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6054e7ae-175c-460b-8de7-6c27cbf501a3-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192502 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192366    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6gj\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-kube-api-access-sf6gj\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192502 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192429    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192992 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192574    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-credential-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192992 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192591    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-certs\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.192992 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.192675    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-socket\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.193204 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.193137    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-data\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.193384 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.193361    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6054e7ae-175c-460b-8de7-6c27cbf501a3-istiod-ca-cert\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.194517 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.194493    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-envoy\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.194737 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.194719    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-podinfo\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.211044 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.211019    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6gj\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-kube-api-access-sf6gj\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.212398 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.212375    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-token\") pod \"e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") " pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.345702 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.345600    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:09.690501 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.689170    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"]
Oct 30 22:04:09.692064 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:04:09.692032    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6054e7ae_175c_460b_8de7_6c27cbf501a3.slice/crio-374cbf2e2c502b88b002ab751d37faa60dedb8604cf82da179d15c5f78bdbcae WatchSource:0}: Error finding container 374cbf2e2c502b88b002ab751d37faa60dedb8604cf82da179d15c5f78bdbcae: Status 404 returned error can't find the container with id 374cbf2e2c502b88b002ab751d37faa60dedb8604cf82da179d15c5f78bdbcae
Oct 30 22:04:09.695941 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.695904    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:09.696074 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.695974    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:09.696074 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:09.696002    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:10.223395 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.223359    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" event={"ID":"6054e7ae-175c-460b-8de7-6c27cbf501a3","Type":"ContainerStarted","Data":"e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3"}
Oct 30 22:04:10.223763 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.223418    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" event={"ID":"6054e7ae-175c-460b-8de7-6c27cbf501a3","Type":"ContainerStarted","Data":"374cbf2e2c502b88b002ab751d37faa60dedb8604cf82da179d15c5f78bdbcae"}
Oct 30 22:04:10.248633 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.248582    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz" podUID="e7c8c5dc-b820-4d48-84c9-8f65e292689a" containerName="payload-processing" containerID="cri-o://d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d" gracePeriod=25
Oct 30 22:04:10.273070 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.272996    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podStartSLOduration=2.272978271 podStartE2EDuration="2.272978271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:10.265966266 +0000 UTC m=+3250.395217725" watchObservedRunningTime="2026-10-30 22:04:10.272978271 +0000 UTC m=+3250.402229730"
Oct 30 22:04:10.346665 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.346630    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:10.348378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.348350    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:04:10.348525 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.348406    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:04:10.549962 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.549934    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:04:10.651648 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.651614    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-kube-api-access-5278d\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-kube-api-access-5278d\") pod \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") "
Oct 30 22:04:10.651841 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.651711    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-plugins-config-volume\") pod \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") "
Oct 30 22:04:10.651841 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.651798    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-routing-overlay-volume\") pod \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") "
Oct 30 22:04:10.652059 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.652042    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-provider-credentials\") pod \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\" (UID: \"e7c8c5dc-b820-4d48-84c9-8f65e292689a\") "
Oct 30 22:04:10.652125 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.652073    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-plugins-config-volume" pod "e7c8c5dc-b820-4d48-84c9-8f65e292689a" (UID: "e7c8c5dc-b820-4d48-84c9-8f65e292689a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:10.652218 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.652118    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-routing-overlay-volume" pod "e7c8c5dc-b820-4d48-84c9-8f65e292689a" (UID: "e7c8c5dc-b820-4d48-84c9-8f65e292689a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:10.652548 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.652528    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:04:10.652672 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.652550    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7c8c5dc-b820-4d48-84c9-8f65e292689a-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:04:10.654131 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.654106    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-kube-api-access-5278d" pod "e7c8c5dc-b820-4d48-84c9-8f65e292689a" (UID: "e7c8c5dc-b820-4d48-84c9-8f65e292689a"). InnerVolumeSpecName "kube-api-access-5278d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.654249 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.654135    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-provider-credentials" pod "e7c8c5dc-b820-4d48-84c9-8f65e292689a" (UID: "e7c8c5dc-b820-4d48-84c9-8f65e292689a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.754650 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.754512    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5278d\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-kube-api-access-5278d\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:04:10.754650 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:10.754545    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7c8c5dc-b820-4d48-84c9-8f65e292689a-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:04:11.032811 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:04:11.032720    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:04:11.156567 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.156535    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:04:11.239835 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.239784    5886 generic.go:356] "Generic (PLEG): container finished" podID="e7c8c5dc-b820-4d48-84c9-8f65e292689a" containerID="d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d" exitCode=0
Oct 30 22:04:11.240279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.239847    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"
Oct 30 22:04:11.240279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.239877    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz" event={"ID":"e7c8c5dc-b820-4d48-84c9-8f65e292689a","Type":"ContainerDied","Data":"d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d"}
Oct 30 22:04:11.240279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.239953    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz" event={"ID":"e7c8c5dc-b820-4d48-84c9-8f65e292689a","Type":"ContainerDied","Data":"18be67fda1e2c8273cd0691eb0b33f1e1dc872333d019481debb54e0054eeb2d"}
Oct 30 22:04:11.240279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.239986    5886 scope.go:122] "RemoveContainer" containerID="d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d"
Oct 30 22:04:11.252593 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.252575    5886 scope.go:122] "RemoveContainer" containerID="d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d"
Oct 30 22:04:11.252864 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:04:11.252844    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d\": container with ID starting with d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d not found: ID does not exist" containerID="d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d"
Oct 30 22:04:11.252962 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.252873    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d"} err="failed to get container status \"d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d\": rpc error: code = NotFound desc = could not find container \"d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d\": container with ID starting with d5f334c2760bf67ea35c35ce18b25b14095c10480c28f01dc952f79e782f7f5d not found: ID does not exist"
Oct 30 22:04:11.306559 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.306436    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"]
Oct 30 22:04:11.311172 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.311133    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-db1288/payload-processing-external-model-e2e-extmod-b-db1288-6b94znbpz"]
Oct 30 22:04:11.346506 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.346475    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:04:11.346668 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:11.346527    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:04:12.014363 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:12.014328    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c8c5dc-b820-4d48-84c9-8f65e292689a" path="/var/lib/kubelet/pods/e7c8c5dc-b820-4d48-84c9-8f65e292689a/volumes"
Oct 30 22:04:12.346255 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:12.346178    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:04:12.346255 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:12.346231    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:04:13.346986 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:13.346951    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:04:13.347408 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:13.347007    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:04:14.346631 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:14.346598    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:04:14.346925 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:14.346650    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:04:15.346824 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.346671    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.354679 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.354631    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:04:15.409371 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.409319    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"]
Oct 30 22:04:15.410490 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.410466    5886 cpu_manager.go:414] "containerMap: removing container" podUID="e7c8c5dc-b820-4d48-84c9-8f65e292689a" containerName="payload-processing"
Oct 30 22:04:15.410572 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.410492    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c8c5dc-b820-4d48-84c9-8f65e292689a" containerName="payload-processing"
Oct 30 22:04:15.410706 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.410693    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c8c5dc-b820-4d48-84c9-8f65e292689a" containerName="payload-processing"
Oct 30 22:04:15.416444 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.416421    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.427028 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.426139    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"]
Oct 30 22:04:15.433376 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.433354    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fdb168-openshift-default-dockercfg-9mvfb\""
Oct 30 22:04:15.459659 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.459633    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-credential-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.459816 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.459689    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istiod-ca-cert\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.459816 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.459783    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cz5n\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-kube-api-access-2cz5n\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.459944 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.459818    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-certs\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.459999 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.459959    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-data\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.460080 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.460055    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-envoy\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.460262 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.460235    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-token\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.460341 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.460287    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.460398 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.460367    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-podinfo\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566072    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-data\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566187    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-envoy\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566261    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-token\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566298    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566340    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-podinfo\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566413    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-credential-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566462    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istiod-ca-cert\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566499    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cz5n\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-kube-api-access-2cz5n\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.566526    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-certs\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.567006    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-certs\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.567274    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-data\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.567499    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-credential-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.568182 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.568127    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istiod-ca-cert\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.569055 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.568412    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-socket\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.588810 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.588774    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-podinfo\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.590030 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.589985    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-envoy\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.600473 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.600367    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-token\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.601304 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.601284    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cz5n\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-kube-api-access-2cz5n\") pod \"e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") " pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:15.733852 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:15.733820    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:16.103309 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.103236    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"]
Oct 30 22:04:16.105628 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:04:16.105597    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ef2a33_4d03_455e_b374_6b7f7224e4b1.slice/crio-c101b9aad948aaeb85e7f167f648289ee11e6e7e9671eec7d009d9a9cf359bb6 WatchSource:0}: Error finding container c101b9aad948aaeb85e7f167f648289ee11e6e7e9671eec7d009d9a9cf359bb6: Status 404 returned error can't find the container with id c101b9aad948aaeb85e7f167f648289ee11e6e7e9671eec7d009d9a9cf359bb6
Oct 30 22:04:16.108932 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.108896    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:16.109012 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.108992    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:16.109055 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.109039    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:16.324502 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.324464    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" event={"ID":"59ef2a33-4d03-455e-b374-6b7f7224e4b1","Type":"ContainerStarted","Data":"2f53d910a5d0c01e669fa7ef6f649cd678bcfb37cc4a6ac11d9102656fd8c7a5"}
Oct 30 22:04:16.324719 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.324520    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" event={"ID":"59ef2a33-4d03-455e-b374-6b7f7224e4b1","Type":"ContainerStarted","Data":"c101b9aad948aaeb85e7f167f648289ee11e6e7e9671eec7d009d9a9cf359bb6"}
Oct 30 22:04:16.346964 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.346930    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:04:16.347405 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.346986    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 22:04:16.382357 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.382231    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podStartSLOduration=1.382212322 podStartE2EDuration="1.382212322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:16.368756989 +0000 UTC m=+3256.498008446" watchObservedRunningTime="2026-10-30 22:04:16.382212322 +0000 UTC m=+3256.511463780"
Oct 30 22:04:16.734786 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:16.734753    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:17.355889 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:17.355855    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:17.356391 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:17.356329    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:17.357115 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:17.357095    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:04:17.736558 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:17.736516    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.736749 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:17.736582    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:04:18.016169 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.016069    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"]
Oct 30 22:04:18.022953 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.022928    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.040823 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.040794    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"]
Oct 30 22:04:18.047672 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.047647    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-ae6492-openshift-default-dockercfg-xlpqc\""
Oct 30 22:04:18.138807 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.138772    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpn5t\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-kube-api-access-jpn5t\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.138980 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.138901    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.139023 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.138992    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.139065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.139045    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.139105 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.139077    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.139105 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.139093    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.139255 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.139240    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.139295 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.139283    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.139372 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.139358    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240308 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240271    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240491 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240324    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240491 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240354    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240491 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240369    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240491 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240464    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240662 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240526    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240662 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240576    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240662 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240629    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpn5t\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-kube-api-access-jpn5t\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240771 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240669    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-credential-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240771 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240723    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-certs\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.240771 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.240758    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.241035 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.241010    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-data\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.241280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.241259    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-socket\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.241487 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.241467    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istiod-ca-cert\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.242827 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.242801    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-envoy\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.243065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.243049    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-podinfo\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.258638 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.258610    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-token\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.258895 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.258878    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpn5t\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-kube-api-access-jpn5t\") pod \"e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") " pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.339886 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.339802    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:18.708889 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.708854    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"]
Oct 30 22:04:18.710888 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:04:18.710860    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e7bf60_4cf6_4ce8_bfc2_7f6cea4f559d.slice/crio-ec570d253abfa9f11b09d5b3a5847f74c4629f3862b17fd10fc4da72563c088c WatchSource:0}: Error finding container ec570d253abfa9f11b09d5b3a5847f74c4629f3862b17fd10fc4da72563c088c: Status 404 returned error can't find the container with id ec570d253abfa9f11b09d5b3a5847f74c4629f3862b17fd10fc4da72563c088c
Oct 30 22:04:18.714429 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.714397    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:18.714511 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.714472    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:18.714548 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.714514    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:04:18.734144 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.734120    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:04:18.734256 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:18.734209    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:04:19.348676 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:19.348636    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" event={"ID":"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d","Type":"ContainerStarted","Data":"cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b"}
Oct 30 22:04:19.348880 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:19.348692    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" event={"ID":"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d","Type":"ContainerStarted","Data":"ec570d253abfa9f11b09d5b3a5847f74c4629f3862b17fd10fc4da72563c088c"}
Oct 30 22:04:19.423911 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:19.423843    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" podStartSLOduration=2.423829463 podStartE2EDuration="2.423829463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:19.407799807 +0000 UTC m=+3259.537051264" watchObservedRunningTime="2026-10-30 22:04:19.423829463 +0000 UTC m=+3259.553080919"
Oct 30 22:04:19.734630 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:19.734586    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.735023 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:19.734658    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:04:20.341053 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:20.341020    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:20.342640 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:20.342607    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:04:20.342797 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:20.342670    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:04:20.735918 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:20.735881    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:04:20.736578 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:20.736540    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:04:21.342561 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:21.342522    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.342879 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:21.342850    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:04:21.734255 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:21.734220    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.734467 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:21.734280    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:04:22.360205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:22.340639    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:04:22.360205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:22.340697    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:04:22.736483 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:22.736434    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:04:22.736697 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:22.736503    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 22:04:23.343645 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:23.343603    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:04:23.343958 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:23.343927    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:04:24.165177 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:24.137187    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:24.165177 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:24.138826    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:24.223181 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:24.220704    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:04:24.344319 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:24.340278    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:04:24.344319 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:24.340347    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:04:25.348803 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:25.348767    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.349429 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:25.349397    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 22:04:26.450113 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:26.450073    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:26.451025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:26.451001    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:26.451970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:26.451949    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:04:37.313523 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.313488    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"]
Oct 30 22:04:37.318266 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.318244    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.338891 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.338864    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"]
Oct 30 22:04:37.346468 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.346439    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8d53e0-serving-cert\""
Oct 30 22:04:37.460936 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.460906    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-metrics-certs\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.461114 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.461050    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.461245 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.461227    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8l2\" (UniqueName: \"kubernetes.io/projected/ff58ecb3-8871-46a9-aa79-a5067e38bb98-kube-api-access-hc8l2\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.461335 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.461320    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff58ecb3-8871-46a9-aa79-a5067e38bb98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.580379 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.571236    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8l2\" (UniqueName: \"kubernetes.io/projected/ff58ecb3-8871-46a9-aa79-a5067e38bb98-kube-api-access-hc8l2\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.580771 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.580736    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff58ecb3-8871-46a9-aa79-a5067e38bb98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.580969 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.580951    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-metrics-certs\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.581231 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.581208    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.595372 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.595337    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff58ecb3-8871-46a9-aa79-a5067e38bb98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.601278 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.601245    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.622747 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.621824    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8l2\" (UniqueName: \"kubernetes.io/projected/ff58ecb3-8871-46a9-aa79-a5067e38bb98-kube-api-access-hc8l2\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.629303 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.629273    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-metrics-certs\") pod \"maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.633507 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.633464    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:37.963971 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.963934    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"]
Oct 30 22:04:37.968761 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:04:37.968718    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff58ecb3_8871_46a9_aa79_a5067e38bb98.slice/crio-c7df1216aa9ab407caeabf470e9a757a69e48aa6304c3bffd939cbdd1c45ff39 WatchSource:0}: Error finding container c7df1216aa9ab407caeabf470e9a757a69e48aa6304c3bffd939cbdd1c45ff39: Status 404 returned error can't find the container with id c7df1216aa9ab407caeabf470e9a757a69e48aa6304c3bffd939cbdd1c45ff39
Oct 30 22:04:37.970542 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:37.970517    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:04:38.694768 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:38.694727    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg" event={"ID":"ff58ecb3-8871-46a9-aa79-a5067e38bb98","Type":"ContainerStarted","Data":"26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08"}
Oct 30 22:04:38.695288 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:38.694798    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg" event={"ID":"ff58ecb3-8871-46a9-aa79-a5067e38bb98","Type":"ContainerStarted","Data":"c7df1216aa9ab407caeabf470e9a757a69e48aa6304c3bffd939cbdd1c45ff39"}
Oct 30 22:04:38.695288 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:38.694828    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:38.735524 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:38.735468    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg" podStartSLOduration=1.173867278 podStartE2EDuration="1.735454399s" totalImagesPullingTime="561.587121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:38.731641215 +0000 UTC m=+3278.860892675" watchObservedRunningTime="2026-10-30 22:04:38.735454399 +0000 UTC m=+3278.864705863"
Oct 30 22:04:41.030946 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:04:41.030906    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:04:43.811745 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:43.811711    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"]
Oct 30 22:04:43.816388 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:43.816365    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:43.833127 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:43.833102    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"]
Oct 30 22:04:43.839561 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:43.839534    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a42c31-serving-cert\""
Oct 30 22:04:43.981360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:43.981328    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:43.981569 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:43.981369    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4pd\" (UniqueName: \"kubernetes.io/projected/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-kube-api-access-4w4pd\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:43.981569 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:43.981534    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-maas-api-tls\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:43.981785 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:43.981767    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-metrics-certs\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.082874 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.082771    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.082874 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.082834    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4pd\" (UniqueName: \"kubernetes.io/projected/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-kube-api-access-4w4pd\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.083134 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.082937    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-maas-api-tls\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.083134 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.083075    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-metrics-certs\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.083657 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.083628    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.085776 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.085752    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-metrics-certs\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.085912 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.085861    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-maas-api-tls\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.102239 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.102209    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4pd\" (UniqueName: \"kubernetes.io/projected/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-kube-api-access-4w4pd\") pod \"maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.131322 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.131294    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:44.447294 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.447263    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"]
Oct 30 22:04:44.449115 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:04:44.449088    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745272f1_fc0f_45ee_9e94_c5c9fc2583ed.slice/crio-ede7f30ef47408c319dff06f4a5fdefa144802e0934bf1205c8e40a27a3642b4 WatchSource:0}: Error finding container ede7f30ef47408c319dff06f4a5fdefa144802e0934bf1205c8e40a27a3642b4: Status 404 returned error can't find the container with id ede7f30ef47408c319dff06f4a5fdefa144802e0934bf1205c8e40a27a3642b4
Oct 30 22:04:44.729884 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.729804    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:04:44.742542 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:44.742511    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m" event={"ID":"745272f1-fc0f-45ee-9e94-c5c9fc2583ed","Type":"ContainerStarted","Data":"ede7f30ef47408c319dff06f4a5fdefa144802e0934bf1205c8e40a27a3642b4"}
Oct 30 22:04:45.782567 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:45.782517    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m" event={"ID":"745272f1-fc0f-45ee-9e94-c5c9fc2583ed","Type":"ContainerStarted","Data":"526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119"}
Oct 30 22:04:45.783738 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:45.783711    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:45.825413 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:45.825337    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m" podStartSLOduration=2.48666738 podStartE2EDuration="2.825317278s" totalImagesPullingTime="338.649898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:45.822853143 +0000 UTC m=+3285.952104601" watchObservedRunningTime="2026-10-30 22:04:45.825317278 +0000 UTC m=+3285.954568736"
Oct 30 22:04:51.996186 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:51.996108    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"]
Oct 30 22:04:51.996738 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:51.996404    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c" podUID="fff50fba-fab4-498f-bd01-8613f3e02a9d" containerName="maas-api" containerID="cri-o://2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886" gracePeriod=30
Oct 30 22:04:52.418550 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.418479    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 22:04:52.490020 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.489984    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-maas-api-tls\") pod \"fff50fba-fab4-498f-bd01-8613f3e02a9d\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") "
Oct 30 22:04:52.490851 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.490825    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff50fba-fab4-498f-bd01-8613f3e02a9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff50fba-fab4-498f-bd01-8613f3e02a9d-openshift-service-ca\") pod \"fff50fba-fab4-498f-bd01-8613f3e02a9d\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") "
Oct 30 22:04:52.491740 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.491713    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff50fba-fab4-498f-bd01-8613f3e02a9d-kube-api-access-2xvnx\" (UniqueName: \"kubernetes.io/projected/fff50fba-fab4-498f-bd01-8613f3e02a9d-kube-api-access-2xvnx\") pod \"fff50fba-fab4-498f-bd01-8613f3e02a9d\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") "
Oct 30 22:04:52.491955 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.491932    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-metrics-certs\") pod \"fff50fba-fab4-498f-bd01-8613f3e02a9d\" (UID: \"fff50fba-fab4-498f-bd01-8613f3e02a9d\") "
Oct 30 22:04:52.492958 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.491397    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff50fba-fab4-498f-bd01-8613f3e02a9d-openshift-service-ca" pod "fff50fba-fab4-498f-bd01-8613f3e02a9d" (UID: "fff50fba-fab4-498f-bd01-8613f3e02a9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:52.528180 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.525728    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff50fba-fab4-498f-bd01-8613f3e02a9d-kube-api-access-2xvnx" pod "fff50fba-fab4-498f-bd01-8613f3e02a9d" (UID: "fff50fba-fab4-498f-bd01-8613f3e02a9d"). InnerVolumeSpecName "kube-api-access-2xvnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:52.578180 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.562297    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-maas-api-tls" pod "fff50fba-fab4-498f-bd01-8613f3e02a9d" (UID: "fff50fba-fab4-498f-bd01-8613f3e02a9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:52.578180 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.562398    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-metrics-certs" pod "fff50fba-fab4-498f-bd01-8613f3e02a9d" (UID: "fff50fba-fab4-498f-bd01-8613f3e02a9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:52.606175 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.605703    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xvnx\" (UniqueName: \"kubernetes.io/projected/fff50fba-fab4-498f-bd01-8613f3e02a9d-kube-api-access-2xvnx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:04:52.606175 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.605741    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:04:52.606175 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.605756    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff50fba-fab4-498f-bd01-8613f3e02a9d-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:04:52.606175 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.605813    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff50fba-fab4-498f-bd01-8613f3e02a9d-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:04:52.803984 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.803953    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:04:52.842654 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.842612    5886 generic.go:356] "Generic (PLEG): container finished" podID="fff50fba-fab4-498f-bd01-8613f3e02a9d" containerID="2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886" exitCode=0
Oct 30 22:04:52.842654 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.842659    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"
Oct 30 22:04:52.842906 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.842686    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c" event={"ID":"fff50fba-fab4-498f-bd01-8613f3e02a9d","Type":"ContainerDied","Data":"2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886"}
Oct 30 22:04:52.842906 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.842737    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c" event={"ID":"fff50fba-fab4-498f-bd01-8613f3e02a9d","Type":"ContainerDied","Data":"fbb90ebfcf4c89d7f3e9bbc6cf8b928a634280f06842605a18aa156fcb5aefb4"}
Oct 30 22:04:52.842906 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.842759    5886 scope.go:122] "RemoveContainer" containerID="2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886"
Oct 30 22:04:52.911088 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.907946    5886 scope.go:122] "RemoveContainer" containerID="2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886"
Oct 30 22:04:52.911088 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:04:52.909076    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886\": container with ID starting with 2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886 not found: ID does not exist" containerID="2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886"
Oct 30 22:04:52.911088 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.909109    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886"} err="failed to get container status \"2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886\": rpc error: code = NotFound desc = could not find container \"2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886\": container with ID starting with 2175d533254d19a1c7ba3edae1bb84284ce65bd93dc176719c410329454af886 not found: ID does not exist"
Oct 30 22:04:52.931057 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.931028    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"]
Oct 30 22:04:52.936380 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:52.936347    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-427a6e-8d4d9bbcc-pgl8c"]
Oct 30 22:04:53.992954 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:04:53.992921    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff50fba-fab4-498f-bd01-8613f3e02a9d" path="/var/lib/kubelet/pods/fff50fba-fab4-498f-bd01-8613f3e02a9d/volumes"
Oct 30 22:05:11.032782 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:05:11.032746    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:05:22.103045 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:22.103008    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"]
Oct 30 22:05:27.103780 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.103726    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7" podUID="1a2e229b-c0f6-4b75-9fc8-8f900efea735" containerName="payload-processing" containerID="cri-o://b0aab307d09860459125102db6a10a6a8e523fa63bce9adad7677312e913bb80" gracePeriod=25
Oct 30 22:05:27.231639 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.231602    5886 generic.go:356] "Generic (PLEG): container finished" podID="1a2e229b-c0f6-4b75-9fc8-8f900efea735" containerID="b0aab307d09860459125102db6a10a6a8e523fa63bce9adad7677312e913bb80" exitCode=0
Oct 30 22:05:27.231780 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.231685    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7" event={"ID":"1a2e229b-c0f6-4b75-9fc8-8f900efea735","Type":"ContainerDied","Data":"b0aab307d09860459125102db6a10a6a8e523fa63bce9adad7677312e913bb80"}
Oct 30 22:05:27.452568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.452544    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:05:27.508791 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.508371    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-provider-credentials\") pod \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") "
Oct 30 22:05:27.509094 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.509070    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-kube-api-access-6wcpx\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-kube-api-access-6wcpx\") pod \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") "
Oct 30 22:05:27.509706 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.509686    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-plugins-config-volume\") pod \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") "
Oct 30 22:05:27.510291 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.510271    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-routing-overlay-volume\") pod \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\" (UID: \"1a2e229b-c0f6-4b75-9fc8-8f900efea735\") "
Oct 30 22:05:27.510961 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.510194    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-plugins-config-volume" pod "1a2e229b-c0f6-4b75-9fc8-8f900efea735" (UID: "1a2e229b-c0f6-4b75-9fc8-8f900efea735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:27.511070 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.510931    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-routing-overlay-volume" pod "1a2e229b-c0f6-4b75-9fc8-8f900efea735" (UID: "1a2e229b-c0f6-4b75-9fc8-8f900efea735"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:27.511624 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.511602    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:27.511624 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.511622    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1a2e229b-c0f6-4b75-9fc8-8f900efea735-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:27.514044 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.514014    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-provider-credentials" pod "1a2e229b-c0f6-4b75-9fc8-8f900efea735" (UID: "1a2e229b-c0f6-4b75-9fc8-8f900efea735"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:27.514128 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.514114    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-kube-api-access-6wcpx" pod "1a2e229b-c0f6-4b75-9fc8-8f900efea735" (UID: "1a2e229b-c0f6-4b75-9fc8-8f900efea735"). InnerVolumeSpecName "kube-api-access-6wcpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:27.638854 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.613847    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:27.638854 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:27.636898    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcpx\" (UniqueName: \"kubernetes.io/projected/1a2e229b-c0f6-4b75-9fc8-8f900efea735-kube-api-access-6wcpx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:28.240874 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:28.240833    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7" event={"ID":"1a2e229b-c0f6-4b75-9fc8-8f900efea735","Type":"ContainerDied","Data":"0340560e8382b6d9e7e20baca2acf0a18ec22bfc6bd0be0c0443d83dd8206913"}
Oct 30 22:05:28.241390 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:28.240905    5886 scope.go:122] "RemoveContainer" containerID="b0aab307d09860459125102db6a10a6a8e523fa63bce9adad7677312e913bb80"
Oct 30 22:05:28.241390 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:28.240847    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"
Oct 30 22:05:28.293146 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:28.293055    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"]
Oct 30 22:05:28.304399 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:28.304366    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ed0866/payload-processing-external-model-e2e-extmod-b-ed0866-87bbg42s7"]
Oct 30 22:05:29.992099 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:29.992056    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2e229b-c0f6-4b75-9fc8-8f900efea735" path="/var/lib/kubelet/pods/1a2e229b-c0f6-4b75-9fc8-8f900efea735/volumes"
Oct 30 22:05:35.550621 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.550575    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"]
Oct 30 22:05:35.551468 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.551441    5886 cpu_manager.go:414] "containerMap: removing container" podUID="fff50fba-fab4-498f-bd01-8613f3e02a9d" containerName="maas-api"
Oct 30 22:05:35.551468 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.551463    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff50fba-fab4-498f-bd01-8613f3e02a9d" containerName="maas-api"
Oct 30 22:05:35.551654 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.551521    5886 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e229b-c0f6-4b75-9fc8-8f900efea735" containerName="payload-processing"
Oct 30 22:05:35.551654 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.551530    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e229b-c0f6-4b75-9fc8-8f900efea735" containerName="payload-processing"
Oct 30 22:05:35.551744 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.551696    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff50fba-fab4-498f-bd01-8613f3e02a9d" containerName="maas-api"
Oct 30 22:05:35.551744 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.551722    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e229b-c0f6-4b75-9fc8-8f900efea735" containerName="payload-processing"
Oct 30 22:05:35.556457 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.556421    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.584501 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.584468    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"]
Oct 30 22:05:35.616956 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.616894    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19a725-openshift-default-dockercfg-jt6v5\""
Oct 30 22:05:35.656510 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.656474    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-token\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.656690 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.656548    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-socket\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.656735 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.656695    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-envoy\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.656772 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.656754    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d282f4-907f-4151-af9b-d5243ddc00ea-istiod-ca-cert\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.656903 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.656885    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqvs\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-kube-api-access-jbqvs\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.656946 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.656917    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-podinfo\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.657087 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.657072    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-credential-socket\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.657144 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.657129    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-data\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.657201 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.657175    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-certs\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.758897 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.758853    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-envoy\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.758907    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d282f4-907f-4151-af9b-d5243ddc00ea-istiod-ca-cert\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.758957    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqvs\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-kube-api-access-jbqvs\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.758972    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-podinfo\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759311 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759138    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-credential-socket\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759311 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759225    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-data\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759311 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759257    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-certs\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759484 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759419    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-token\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759484 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759473    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-socket\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759605 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759484    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-credential-socket\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759671 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759597    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-data\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759736 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759675    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-certs\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759828 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759789    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d282f4-907f-4151-af9b-d5243ddc00ea-istiod-ca-cert\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.759895 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.759876    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-socket\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.761560 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.761529    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-envoy\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.761702 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.761686    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-podinfo\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.776917 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.776889    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-token\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.777058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.777040    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqvs\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-kube-api-access-jbqvs\") pod \"e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") " pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:35.880260 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:35.876458    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:36.392478 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:36.392451    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"]
Oct 30 22:05:36.395287 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:05:36.395257    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d282f4_907f_4151_af9b_d5243ddc00ea.slice/crio-7f35d0e4dca96b536b69ac54429a96ec361889455033f004813c3b9966902190 WatchSource:0}: Error finding container 7f35d0e4dca96b536b69ac54429a96ec361889455033f004813c3b9966902190: Status 404 returned error can't find the container with id 7f35d0e4dca96b536b69ac54429a96ec361889455033f004813c3b9966902190
Oct 30 22:05:36.398707 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:36.398675    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:36.398808 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:36.398740    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:36.398808 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:36.398778    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:05:37.334360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:37.334320    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" event={"ID":"d4d282f4-907f-4151-af9b-d5243ddc00ea","Type":"ContainerStarted","Data":"820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628"}
Oct 30 22:05:37.334867 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:37.334385    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" event={"ID":"d4d282f4-907f-4151-af9b-d5243ddc00ea","Type":"ContainerStarted","Data":"7f35d0e4dca96b536b69ac54429a96ec361889455033f004813c3b9966902190"}
Oct 30 22:05:37.395956 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:37.395885    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" podStartSLOduration=2.395866804 podStartE2EDuration="2.395866804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:37.393447988 +0000 UTC m=+3337.522699448" watchObservedRunningTime="2026-10-30 22:05:37.395866804 +0000 UTC m=+3337.525118261"
Oct 30 22:05:37.877228 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:37.877186    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:37.883780 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:37.883747    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:38.351279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:38.350081    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:38.351961 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:38.351886    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:05:41.029660 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:05:41.029581    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:05:41.945177 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:41.939916    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"]
Oct 30 22:05:42.022181 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.012935    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:42.103196 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.102629    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19a725\""
Oct 30 22:05:42.103196 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.102833    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19a725-dockercfg-4vrvt\""
Oct 30 22:05:42.110198 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.110168    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"]
Oct 30 22:05:42.133822 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.133787    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9\" (UID: \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:42.134030 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.133838    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52656\" (UniqueName: \"kubernetes.io/projected/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-kube-api-access-52656\") pod \"payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9\" (UID: \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:42.235602 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.235512    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9\" (UID: \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:42.235602 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.235551    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52656\" (UniqueName: \"kubernetes.io/projected/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-kube-api-access-52656\") pod \"payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9\" (UID: \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:42.236291 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.236269    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9\" (UID: \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:42.256191 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.256134    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52656\" (UniqueName: \"kubernetes.io/projected/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-kube-api-access-52656\") pod \"payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9\" (UID: \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:42.417252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.417218    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:42.731301 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:42.731268    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"]
Oct 30 22:05:42.735357 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:05:42.735324    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21651e7e_fce1_4e60_b50a_20b0ce4f77b9.slice/crio-ae3b6fa1e101d1d195612616b8f49817a32d5085732498aaef1232979e24ecc1 WatchSource:0}: Error finding container ae3b6fa1e101d1d195612616b8f49817a32d5085732498aaef1232979e24ecc1: Status 404 returned error can't find the container with id ae3b6fa1e101d1d195612616b8f49817a32d5085732498aaef1232979e24ecc1
Oct 30 22:05:43.422313 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:43.422099    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9" event={"ID":"21651e7e-fce1-4e60-b50a-20b0ce4f77b9","Type":"ContainerStarted","Data":"9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e"}
Oct 30 22:05:43.422313 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:43.422188    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9" event={"ID":"21651e7e-fce1-4e60-b50a-20b0ce4f77b9","Type":"ContainerStarted","Data":"ae3b6fa1e101d1d195612616b8f49817a32d5085732498aaef1232979e24ecc1"}
Oct 30 22:05:43.422313 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:43.422255    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:43.463447 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:43.463388    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9" podStartSLOduration=2.463372523 podStartE2EDuration="2.463372523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:43.458852444 +0000 UTC m=+3343.588103901" watchObservedRunningTime="2026-10-30 22:05:43.463372523 +0000 UTC m=+3343.592623981"
Oct 30 22:05:46.445581 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:46.445542    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"]
Oct 30 22:05:46.446070 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:46.445776    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws" podUID="5f6871c7-ebe8-457e-b8af-e2025076b1a3" containerName="maas-api" containerID="cri-o://bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac" gracePeriod=30
Oct 30 22:05:47.039321 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.038892    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 22:05:47.156368 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.156321    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6871c7-ebe8-457e-b8af-e2025076b1a3-kube-api-access-mk5qq\" (UniqueName: \"kubernetes.io/projected/5f6871c7-ebe8-457e-b8af-e2025076b1a3-kube-api-access-mk5qq\") pod \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") "
Oct 30 22:05:47.156549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.156481    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls\") pod \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") "
Oct 30 22:05:47.156622 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.156607    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6871c7-ebe8-457e-b8af-e2025076b1a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6871c7-ebe8-457e-b8af-e2025076b1a3-openshift-service-ca\") pod \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") "
Oct 30 22:05:47.156686 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.156668    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-metrics-certs\") pod \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\" (UID: \"5f6871c7-ebe8-457e-b8af-e2025076b1a3\") "
Oct 30 22:05:47.157118 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.157083    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6871c7-ebe8-457e-b8af-e2025076b1a3-openshift-service-ca" pod "5f6871c7-ebe8-457e-b8af-e2025076b1a3" (UID: "5f6871c7-ebe8-457e-b8af-e2025076b1a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:47.158817 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.158787    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6871c7-ebe8-457e-b8af-e2025076b1a3-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:47.159097 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.159071    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6871c7-ebe8-457e-b8af-e2025076b1a3-kube-api-access-mk5qq" pod "5f6871c7-ebe8-457e-b8af-e2025076b1a3" (UID: "5f6871c7-ebe8-457e-b8af-e2025076b1a3"). InnerVolumeSpecName "kube-api-access-mk5qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:47.159252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.159101    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-metrics-certs" pod "5f6871c7-ebe8-457e-b8af-e2025076b1a3" (UID: "5f6871c7-ebe8-457e-b8af-e2025076b1a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:47.160478 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.160456    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls" pod "5f6871c7-ebe8-457e-b8af-e2025076b1a3" (UID: "5f6871c7-ebe8-457e-b8af-e2025076b1a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:47.259814 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.259733    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk5qq\" (UniqueName: \"kubernetes.io/projected/5f6871c7-ebe8-457e-b8af-e2025076b1a3-kube-api-access-mk5qq\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:47.259814 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.259763    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:47.259814 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.259775    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f6871c7-ebe8-457e-b8af-e2025076b1a3-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:47.466990 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.466954    5886 generic.go:356] "Generic (PLEG): container finished" podID="5f6871c7-ebe8-457e-b8af-e2025076b1a3" containerID="bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac" exitCode=0
Oct 30 22:05:47.467577 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.467029    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"
Oct 30 22:05:47.467577 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.467070    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws" event={"ID":"5f6871c7-ebe8-457e-b8af-e2025076b1a3","Type":"ContainerDied","Data":"bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac"}
Oct 30 22:05:47.467577 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.467195    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws" event={"ID":"5f6871c7-ebe8-457e-b8af-e2025076b1a3","Type":"ContainerDied","Data":"a2fd1eab9e11937287b88f7c044ca25ff3806d97b4fb3a7a6de89a01bd286bcf"}
Oct 30 22:05:47.467577 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.467235    5886 scope.go:122] "RemoveContainer" containerID="bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac"
Oct 30 22:05:47.483551 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.483530    5886 scope.go:122] "RemoveContainer" containerID="bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac"
Oct 30 22:05:47.483863 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:05:47.483836    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac\": container with ID starting with bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac not found: ID does not exist" containerID="bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac"
Oct 30 22:05:47.483925 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.483875    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac"} err="failed to get container status \"bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac\": rpc error: code = NotFound desc = could not find container \"bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac\": container with ID starting with bcc13df886bc745c562e9178a90469610c4def40892c6ae87de9cff210837fac not found: ID does not exist"
Oct 30 22:05:47.517119 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.517058    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"]
Oct 30 22:05:47.524127 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.524105    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-db1288-dff644984-v9zws"]
Oct 30 22:05:47.990449 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:47.990415    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6871c7-ebe8-457e-b8af-e2025076b1a3" path="/var/lib/kubelet/pods/5f6871c7-ebe8-457e-b8af-e2025076b1a3/volumes"
Oct 30 22:05:54.436461 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:54.436424    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:05:54.595306 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:54.595266    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"]
Oct 30 22:05:54.595858 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:54.595821    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy" containerID="cri-o://8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196" gracePeriod=30
Oct 30 22:05:59.889137 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.889106    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 22:05:59.936236 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936205    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-credential-socket\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936403 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936279    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-token\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-token\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936403 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936349    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-socket\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936403 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936396    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-envoy\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936545 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936445    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-kube-api-access-vddv8\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-kube-api-access-vddv8\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936545 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936516    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-credential-socket" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:59.936668 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936617    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42d676f-8ffd-4663-8771-ec3c17bff079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42d676f-8ffd-4663-8771-ec3c17bff079-istiod-ca-cert\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936668 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936626    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-socket" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:59.936781 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936683    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-data\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936781 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936749    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-podinfo\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936894 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936812    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-certs\") pod \"c42d676f-8ffd-4663-8771-ec3c17bff079\" (UID: \"c42d676f-8ffd-4663-8771-ec3c17bff079\") "
Oct 30 22:05:59.936960 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936939    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-data" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:59.937020 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.936976    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42d676f-8ffd-4663-8771-ec3c17bff079-istiod-ca-cert" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:59.937222 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.937199    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-certs" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:59.937496 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.937476    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c42d676f-8ffd-4663-8771-ec3c17bff079-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:59.937496 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.937497    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:59.937641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.937505    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:59.937641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.937513    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:59.937641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.937523    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:05:59.938680 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.938653    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-token" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:59.938775 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.938760    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-kube-api-access-vddv8" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "kube-api-access-vddv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:59.939106 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.939089    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-podinfo" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:05:59.939191 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:05:59.939101    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-envoy" pod "c42d676f-8ffd-4663-8771-ec3c17bff079" (UID: "c42d676f-8ffd-4663-8771-ec3c17bff079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:00.039057 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.039015    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:00.039057 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.039040    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:00.039057 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.039054    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vddv8\" (UniqueName: \"kubernetes.io/projected/c42d676f-8ffd-4663-8771-ec3c17bff079-kube-api-access-vddv8\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:00.039057 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.039069    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c42d676f-8ffd-4663-8771-ec3c17bff079-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:00.574044 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.574012    5886 generic.go:356] "Generic (PLEG): container finished" podID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerID="8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196" exitCode=0
Oct 30 22:06:00.574264 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.574063    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" event={"ID":"c42d676f-8ffd-4663-8771-ec3c17bff079","Type":"ContainerDied","Data":"8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196"}
Oct 30 22:06:00.574264 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.574087    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"
Oct 30 22:06:00.574264 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.574134    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82" event={"ID":"c42d676f-8ffd-4663-8771-ec3c17bff079","Type":"ContainerDied","Data":"1390a69ec9574a97f0738ab954cc4e7da98ce6655ad560a99526b8642734761e"}
Oct 30 22:06:00.574264 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.574189    5886 scope.go:122] "RemoveContainer" containerID="8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196"
Oct 30 22:06:00.588473 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.588449    5886 scope.go:122] "RemoveContainer" containerID="be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db"
Oct 30 22:06:00.601558 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.601532    5886 scope.go:122] "RemoveContainer" containerID="8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196"
Oct 30 22:06:00.601918 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:06:00.601886    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196\": container with ID starting with 8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196 not found: ID does not exist" containerID="8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196"
Oct 30 22:06:00.602110 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.601929    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196"} err="failed to get container status \"8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196\": rpc error: code = NotFound desc = could not find container \"8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196\": container with ID starting with 8ebcafd59ed7f553cb4157a6494350d7a00398b7346c4b2a8f9c7079806ed196 not found: ID does not exist"
Oct 30 22:06:00.602110 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.601955    5886 scope.go:122] "RemoveContainer" containerID="be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db"
Oct 30 22:06:00.602397 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:06:00.602333    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db\": container with ID starting with be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db not found: ID does not exist" containerID="be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db"
Oct 30 22:06:00.602397 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.602363    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db"} err="failed to get container status \"be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db\": rpc error: code = NotFound desc = could not find container \"be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db\": container with ID starting with be73980fe76007d4596655aea8454f21a695223deb3d2fe11f64565ff58832db not found: ID does not exist"
Oct 30 22:06:00.633069 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.633038    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"]
Oct 30 22:06:00.644621 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:00.644594    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-db1288-openshift-default-55fffb9d9-f5d82"]
Oct 30 22:06:01.992017 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:01.991985    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" path="/var/lib/kubelet/pods/c42d676f-8ffd-4663-8771-ec3c17bff079/volumes"
Oct 30 22:06:11.030910 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:06:11.030868    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:06:14.327715 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.327681    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"]
Oct 30 22:06:14.328593 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.328573    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy"
Oct 30 22:06:14.328672 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.328595    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy"
Oct 30 22:06:14.328672 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.328615    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy"
Oct 30 22:06:14.328672 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.328621    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy"
Oct 30 22:06:14.328672 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.328644    5886 cpu_manager.go:414] "containerMap: removing container" podUID="5f6871c7-ebe8-457e-b8af-e2025076b1a3" containerName="maas-api"
Oct 30 22:06:14.328672 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.328649    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6871c7-ebe8-457e-b8af-e2025076b1a3" containerName="maas-api"
Oct 30 22:06:14.328833 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.328756    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy"
Oct 30 22:06:14.328833 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.328768    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6871c7-ebe8-457e-b8af-e2025076b1a3" containerName="maas-api"
Oct 30 22:06:14.333739 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.333711    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.350366 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.350334    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"]
Oct 30 22:06:14.357335 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.357309    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6537e6-serving-cert\""
Oct 30 22:06:14.386049 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.386010    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz6dw\" (UniqueName: \"kubernetes.io/projected/7ed45761-7d92-4890-9405-3743f30a43bb-kube-api-access-hz6dw\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.386243 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.386172    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-metrics-certs\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.386304 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.386267    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-maas-api-tls\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.386346 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.386302    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed45761-7d92-4890-9405-3743f30a43bb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.487447 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.487413    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz6dw\" (UniqueName: \"kubernetes.io/projected/7ed45761-7d92-4890-9405-3743f30a43bb-kube-api-access-hz6dw\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.487660 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.487474    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-metrics-certs\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.487660 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.487522    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-maas-api-tls\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.487660 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.487541    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed45761-7d92-4890-9405-3743f30a43bb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.488501 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.488468    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed45761-7d92-4890-9405-3743f30a43bb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.490306 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.490279    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-metrics-certs\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.490429 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.490279    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-maas-api-tls\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.506937 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.506900    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz6dw\" (UniqueName: \"kubernetes.io/projected/7ed45761-7d92-4890-9405-3743f30a43bb-kube-api-access-hz6dw\") pod \"maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.648805 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.648296    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:14.962068 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:14.961991    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"]
Oct 30 22:06:14.966114 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:06:14.966086    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed45761_7d92_4890_9405_3743f30a43bb.slice/crio-788075af98ecea530ceb40e44e8c08ff3521184cfd384ce72074368cfc6f5c52 WatchSource:0}: Error finding container 788075af98ecea530ceb40e44e8c08ff3521184cfd384ce72074368cfc6f5c52: Status 404 returned error can't find the container with id 788075af98ecea530ceb40e44e8c08ff3521184cfd384ce72074368cfc6f5c52
Oct 30 22:06:15.734430 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:15.734386    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr" event={"ID":"7ed45761-7d92-4890-9405-3743f30a43bb","Type":"ContainerStarted","Data":"5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333"}
Oct 30 22:06:15.734906 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:15.734453    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr" event={"ID":"7ed45761-7d92-4890-9405-3743f30a43bb","Type":"ContainerStarted","Data":"788075af98ecea530ceb40e44e8c08ff3521184cfd384ce72074368cfc6f5c52"}
Oct 30 22:06:15.734906 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:15.734477    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:15.779879 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:15.779825    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr" podStartSLOduration=1.410143238 podStartE2EDuration="1.779806433s" totalImagesPullingTime="369.663195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:15.776010471 +0000 UTC m=+3375.905261929" watchObservedRunningTime="2026-10-30 22:06:15.779806433 +0000 UTC m=+3375.909057892"
Oct 30 22:06:18.118682 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.118646    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"]
Oct 30 22:06:18.119520 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.119502    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42d676f-8ffd-4663-8771-ec3c17bff079" containerName="istio-proxy"
Oct 30 22:06:18.123500 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.123482    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.141523 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.141497    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"]
Oct 30 22:06:18.155537 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.148279    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5454af-serving-cert\""
Oct 30 22:06:18.217740 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.217693    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.217920 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.217810    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.217920 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.217898    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpmh\" (UniqueName: \"kubernetes.io/projected/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-kube-api-access-fzpmh\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.218113 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.218090    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-metrics-certs\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.323178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.322188    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-metrics-certs\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.323178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.322385    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.323178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.322454    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.323178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.322542    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpmh\" (UniqueName: \"kubernetes.io/projected/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-kube-api-access-fzpmh\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.337217 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.332773    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.337217 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:06:18.332897    5886 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-serving-cert: secret "maas-api-e2e-inf-a-5454af-serving-cert" not found
Oct 30 22:06:18.337217 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:06:18.332966    5886 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls podName:1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e nodeName:}" failed. No retries permitted until 2026-10-30 22:06:18.832948257 +0000 UTC m=+3378.962199705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls") pod "maas-api-e2e-inf-a-5454af-977786845-4hcf7" (UID: "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e") : secret "maas-api-e2e-inf-a-5454af-serving-cert" not found
Oct 30 22:06:18.352181 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.350083    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-metrics-certs\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.354369 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.353668    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpmh\" (UniqueName: \"kubernetes.io/projected/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-kube-api-access-fzpmh\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.931902 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.931862    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:18.934270 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:18.934248    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls\") pod \"maas-api-e2e-inf-a-5454af-977786845-4hcf7\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:19.039592 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:19.039335    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:19.410319 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:19.410263    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"]
Oct 30 22:06:19.423793 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:06:19.423751    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3a4d7f_1c3e_4705_9b8a_bdfa32012f9e.slice/crio-529e382d2ef88f9f9def81dd3914b398316a8f9457d5052d7ef1ce273d0a20ad WatchSource:0}: Error finding container 529e382d2ef88f9f9def81dd3914b398316a8f9457d5052d7ef1ce273d0a20ad: Status 404 returned error can't find the container with id 529e382d2ef88f9f9def81dd3914b398316a8f9457d5052d7ef1ce273d0a20ad
Oct 30 22:06:19.794941 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:19.794905    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7" event={"ID":"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e","Type":"ContainerStarted","Data":"529e382d2ef88f9f9def81dd3914b398316a8f9457d5052d7ef1ce273d0a20ad"}
Oct 30 22:06:20.804556 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:20.804513    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7" event={"ID":"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e","Type":"ContainerStarted","Data":"6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb"}
Oct 30 22:06:20.804988 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:20.804788    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:20.845720 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:20.845668    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7" podStartSLOduration=2.491106019 podStartE2EDuration="2.845653372s" totalImagesPullingTime="354.547353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:20.841367634 +0000 UTC m=+3380.970619093" watchObservedRunningTime="2026-10-30 22:06:20.845653372 +0000 UTC m=+3380.974904873"
Oct 30 22:06:21.763273 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:21.763234    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:06:26.823286 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:26.823215    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:06:27.607597 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:27.607560    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"]
Oct 30 22:06:31.847505 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:31.847469    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"]
Oct 30 22:06:31.847971 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:31.847712    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4" podUID="9dd3d315-e31c-4c1f-a441-465b68dcd02f" containerName="maas-api" containerID="cri-o://df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c" gracePeriod=30
Oct 30 22:06:32.220390 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.220357    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 22:06:32.399960 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.399921    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd3d315-e31c-4c1f-a441-465b68dcd02f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3d315-e31c-4c1f-a441-465b68dcd02f-openshift-service-ca\") pod \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") "
Oct 30 22:06:32.401077 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.401052    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-maas-api-tls\") pod \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") "
Oct 30 22:06:32.401328 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.401310    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd3d315-e31c-4c1f-a441-465b68dcd02f-kube-api-access-dxnlm\" (UniqueName: \"kubernetes.io/projected/9dd3d315-e31c-4c1f-a441-465b68dcd02f-kube-api-access-dxnlm\") pod \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") "
Oct 30 22:06:32.410253 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.410230    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-metrics-certs\") pod \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\" (UID: \"9dd3d315-e31c-4c1f-a441-465b68dcd02f\") "
Oct 30 22:06:32.412392 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.412366    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd3d315-e31c-4c1f-a441-465b68dcd02f-openshift-service-ca" pod "9dd3d315-e31c-4c1f-a441-465b68dcd02f" (UID: "9dd3d315-e31c-4c1f-a441-465b68dcd02f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:32.474235 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.470930    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-metrics-certs" pod "9dd3d315-e31c-4c1f-a441-465b68dcd02f" (UID: "9dd3d315-e31c-4c1f-a441-465b68dcd02f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:32.478996 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.478817    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd3d315-e31c-4c1f-a441-465b68dcd02f-kube-api-access-dxnlm" pod "9dd3d315-e31c-4c1f-a441-465b68dcd02f" (UID: "9dd3d315-e31c-4c1f-a441-465b68dcd02f"). InnerVolumeSpecName "kube-api-access-dxnlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:32.484919 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.484879    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-maas-api-tls" pod "9dd3d315-e31c-4c1f-a441-465b68dcd02f" (UID: "9dd3d315-e31c-4c1f-a441-465b68dcd02f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:32.513010 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.512975    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3d315-e31c-4c1f-a441-465b68dcd02f-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:32.513010 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.513012    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:32.513327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.513026    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxnlm\" (UniqueName: \"kubernetes.io/projected/9dd3d315-e31c-4c1f-a441-465b68dcd02f-kube-api-access-dxnlm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:32.513327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.513039    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd3d315-e31c-4c1f-a441-465b68dcd02f-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:32.607908 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.607835    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf" podUID="2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" containerName="payload-pre-processing" containerID="cri-o://64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de" gracePeriod=25
Oct 30 22:06:32.931342 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.931308    5886 generic.go:356] "Generic (PLEG): container finished" podID="9dd3d315-e31c-4c1f-a441-465b68dcd02f" containerID="df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c" exitCode=0
Oct 30 22:06:32.931822 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.931357    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"
Oct 30 22:06:32.931822 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.931398    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4" event={"ID":"9dd3d315-e31c-4c1f-a441-465b68dcd02f","Type":"ContainerDied","Data":"df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c"}
Oct 30 22:06:32.931822 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.931450    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4" event={"ID":"9dd3d315-e31c-4c1f-a441-465b68dcd02f","Type":"ContainerDied","Data":"1ef05c708f9b686b120d144c8821a148604b51414c7be773ec5de84c0f9c1dd4"}
Oct 30 22:06:32.931822 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.931473    5886 scope.go:122] "RemoveContainer" containerID="df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c"
Oct 30 22:06:32.942737 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.942718    5886 scope.go:122] "RemoveContainer" containerID="df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c"
Oct 30 22:06:32.942985 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:06:32.942966    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c\": container with ID starting with df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c not found: ID does not exist" containerID="df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c"
Oct 30 22:06:32.943049 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.942995    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c"} err="failed to get container status \"df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c\": rpc error: code = NotFound desc = could not find container \"df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c\": container with ID starting with df859c7c384e4f118762e2be9cd06f9b38cd176a198bd4a171399b1f275ab42c not found: ID does not exist"
Oct 30 22:06:32.983553 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.983526    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"]
Oct 30 22:06:32.994186 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:32.994147    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f49db2-56db58c65b-drnc4"]
Oct 30 22:06:33.990500 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:33.990466    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd3d315-e31c-4c1f-a441-465b68dcd02f" path="/var/lib/kubelet/pods/9dd3d315-e31c-4c1f-a441-465b68dcd02f/volumes"
Oct 30 22:06:36.733631 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:36.733590    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf" podUID="2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:41.032659 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:06:41.032613    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:06:45.289793 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:45.289753    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"]
Oct 30 22:06:46.734255 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:46.734213    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf" podUID="2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:52.928077 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:52.928048    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 22:06:52.950191 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:52.950120    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-plugins-config-volume\") pod \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\" (UID: \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\") "
Oct 30 22:06:52.950363 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:52.950273    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-kube-api-access-x75bv\" (UniqueName: \"kubernetes.io/projected/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-kube-api-access-x75bv\") pod \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\" (UID: \"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037\") "
Oct 30 22:06:52.950540 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:52.950515    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-plugins-config-volume" pod "2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" (UID: "2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:52.950976 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:52.950952    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:52.952447 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:52.952423    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-kube-api-access-x75bv" pod "2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" (UID: "2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037"). InnerVolumeSpecName "kube-api-access-x75bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:53.052689 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.052651    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75bv\" (UniqueName: \"kubernetes.io/projected/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037-kube-api-access-x75bv\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:06:53.190816 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.190773    5886 generic.go:356] "Generic (PLEG): container finished" podID="2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" containerID="64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de" exitCode=0
Oct 30 22:06:53.190977 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.190839    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"
Oct 30 22:06:53.190977 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.190849    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf" event={"ID":"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037","Type":"ContainerDied","Data":"64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de"}
Oct 30 22:06:53.190977 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.190916    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf" event={"ID":"2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037","Type":"ContainerDied","Data":"f58f5f402fa74a79e5684d7299905464d8efecf50a40a74ed453906986932b35"}
Oct 30 22:06:53.190977 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.190948    5886 scope.go:122] "RemoveContainer" containerID="64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de"
Oct 30 22:06:53.203900 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.203873    5886 scope.go:122] "RemoveContainer" containerID="64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de"
Oct 30 22:06:53.204254 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:06:53.204234    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de\": container with ID starting with 64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de not found: ID does not exist" containerID="64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de"
Oct 30 22:06:53.204356 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.204274    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de"} err="failed to get container status \"64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de\": rpc error: code = NotFound desc = could not find container \"64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de\": container with ID starting with 64adcdae2989078c088b276be0c5b95807339775f0a45f08f162e37bd93d99de not found: ID does not exist"
Oct 30 22:06:53.242660 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.242598    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"]
Oct 30 22:06:53.246649 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.246620    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f49db2-66bcf46db7-5qkcf"]
Oct 30 22:06:53.989900 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:06:53.989856    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" path="/var/lib/kubelet/pods/2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037/volumes"
Oct 30 22:07:00.321607 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.321542    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" containerID="cri-o://3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328" gracePeriod=45
Oct 30 22:07:00.690717 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.690683    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 22:07:00.829766 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.829612    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-model-cache\") pod \"35fb6198-7dfd-44f0-a674-0d9287320862\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") "
Oct 30 22:07:00.829766 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.829673    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-home\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-home\") pod \"35fb6198-7dfd-44f0-a674-0d9287320862\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") "
Oct 30 22:07:00.830084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.829843    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-dshm\") pod \"35fb6198-7dfd-44f0-a674-0d9287320862\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") "
Oct 30 22:07:00.830084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.829905    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-tmp-dir\") pod \"35fb6198-7dfd-44f0-a674-0d9287320862\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") "
Oct 30 22:07:00.830084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.829950    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-model-cache" pod "35fb6198-7dfd-44f0-a674-0d9287320862" (UID: "35fb6198-7dfd-44f0-a674-0d9287320862"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:00.830084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.830026    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35fb6198-7dfd-44f0-a674-0d9287320862-tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb6198-7dfd-44f0-a674-0d9287320862-tls-certs\") pod \"35fb6198-7dfd-44f0-a674-0d9287320862\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") "
Oct 30 22:07:00.830084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.830054    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-home" pod "35fb6198-7dfd-44f0-a674-0d9287320862" (UID: "35fb6198-7dfd-44f0-a674-0d9287320862"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:00.830474 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.830094    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fb6198-7dfd-44f0-a674-0d9287320862-kube-api-access-ctdln\" (UniqueName: \"kubernetes.io/projected/35fb6198-7dfd-44f0-a674-0d9287320862-kube-api-access-ctdln\") pod \"35fb6198-7dfd-44f0-a674-0d9287320862\" (UID: \"35fb6198-7dfd-44f0-a674-0d9287320862\") "
Oct 30 22:07:00.830474 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.830315    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-tmp-dir" pod "35fb6198-7dfd-44f0-a674-0d9287320862" (UID: "35fb6198-7dfd-44f0-a674-0d9287320862"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:00.830809 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.830789    5886 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:07:00.830889 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.830813    5886 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:07:00.830889 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.830835    5886 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:07:00.836200 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.833251    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fb6198-7dfd-44f0-a674-0d9287320862-kube-api-access-ctdln" pod "35fb6198-7dfd-44f0-a674-0d9287320862" (UID: "35fb6198-7dfd-44f0-a674-0d9287320862"). InnerVolumeSpecName "kube-api-access-ctdln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:00.843343 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.843303    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35fb6198-7dfd-44f0-a674-0d9287320862-tls-certs" pod "35fb6198-7dfd-44f0-a674-0d9287320862" (UID: "35fb6198-7dfd-44f0-a674-0d9287320862"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:00.845295 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.845265    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-dshm" pod "35fb6198-7dfd-44f0-a674-0d9287320862" (UID: "35fb6198-7dfd-44f0-a674-0d9287320862"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:00.932735 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.932701    5886 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35fb6198-7dfd-44f0-a674-0d9287320862-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:07:00.932735 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.932735    5886 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fb6198-7dfd-44f0-a674-0d9287320862-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:07:00.933017 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:00.932752    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdln\" (UniqueName: \"kubernetes.io/projected/35fb6198-7dfd-44f0-a674-0d9287320862-kube-api-access-ctdln\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:07:01.267918 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.267880    5886 generic.go:356] "Generic (PLEG): container finished" podID="35fb6198-7dfd-44f0-a674-0d9287320862" containerID="3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328" exitCode=0
Oct 30 22:07:01.268118 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.268072    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" event={"ID":"35fb6198-7dfd-44f0-a674-0d9287320862","Type":"ContainerDied","Data":"3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328"}
Oct 30 22:07:01.268118 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.268110    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" event={"ID":"35fb6198-7dfd-44f0-a674-0d9287320862","Type":"ContainerDied","Data":"88035ef18dead5f4b287d9c8bb8d03331e682c7a76059c0291949858ddb4a829"}
Oct 30 22:07:01.268306 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.268133    5886 scope.go:122] "RemoveContainer" containerID="3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328"
Oct 30 22:07:01.268418 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.268399    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"
Oct 30 22:07:01.283750 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.283719    5886 scope.go:122] "RemoveContainer" containerID="3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328"
Oct 30 22:07:01.284124 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:07:01.284096    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328\": container with ID starting with 3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328 not found: ID does not exist" containerID="3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328"
Oct 30 22:07:01.284312 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.284137    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328"} err="failed to get container status \"3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328\": rpc error: code = NotFound desc = could not find container \"3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328\": container with ID starting with 3d842f4c9b3c4fbbe7cb5af6e76ff84f5abcb9fc707789af7aea4176b7b38328 not found: ID does not exist"
Oct 30 22:07:01.320722 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.320693    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"]
Oct 30 22:07:01.324195 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.324171    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5"]
Oct 30 22:07:01.667189 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.667112    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-49738b/premium-w0-49738b-kserve-6dc5bb4f59-g8hm5" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:01.991343 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:01.991264    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" path="/var/lib/kubelet/pods/35fb6198-7dfd-44f0-a674-0d9287320862/volumes"
Oct 30 22:07:05.310971 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:05.310934    5886 generic.go:356] "Generic (PLEG): container finished" podID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerID="820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628" exitCode=0
Oct 30 22:07:05.311542 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:05.311011    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" event={"ID":"d4d282f4-907f-4151-af9b-d5243ddc00ea","Type":"ContainerDied","Data":"820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628"}
Oct 30 22:07:05.311542 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:05.311496    5886 scope.go:122] "RemoveContainer" containerID="820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628"
Oct 30 22:07:05.313994 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:05.313953    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:05.314116 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:05.314061    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:05.314116 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:05.314108    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:07:06.320198 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:06.320138    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" event={"ID":"d4d282f4-907f-4151-af9b-d5243ddc00ea","Type":"ContainerStarted","Data":"420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570"}
Oct 30 22:07:06.884528 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:06.884491    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:07:06.892327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:06.892286    5886 patch_prober.go:29] interesting pod/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:07:06.892656 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:06.892625    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 22:07:07.889884 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:07.889849    5886 patch_prober.go:29] interesting pod/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:07:07.890505 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:07.890474    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 22:07:08.350773 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:08.350734    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:07:08.889794 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:08.889758    5886 patch_prober.go:29] interesting pod/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:07:08.890300 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:08.890268    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 22:07:09.880742 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:09.880705    5886 patch_prober.go:29] interesting pod/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:07:09.881018 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:09.880990    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 22:07:10.881330 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:10.881241    5886 patch_prober.go:29] interesting pod/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:07:10.882066 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:10.882029    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 22:07:11.034112 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:07:11.034078    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:07:11.878631 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:11.878570    5886 patch_prober.go:29] interesting pod/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 22:07:11.878959 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:11.878909    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 22:07:12.883224 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:12.883145    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:07:12.884845 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:12.884818    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:07:41.088640 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:07:41.088594    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:07:53.182981 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.182940    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"]
Oct 30 22:07:53.184029 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184009    5886 cpu_manager.go:414] "containerMap: removing container" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main"
Oct 30 22:07:53.184077 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184033    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main"
Oct 30 22:07:53.184077 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184073    5886 cpu_manager.go:414] "containerMap: removing container" podUID="9dd3d315-e31c-4c1f-a441-465b68dcd02f" containerName="maas-api"
Oct 30 22:07:53.184144 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184081    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd3d315-e31c-4c1f-a441-465b68dcd02f" containerName="maas-api"
Oct 30 22:07:53.184144 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184135    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" containerName="payload-pre-processing"
Oct 30 22:07:53.184229 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184142    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" containerName="payload-pre-processing"
Oct 30 22:07:53.184336 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184322    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbb8d0d-a8cb-4b10-9ff0-3b3d32075037" containerName="payload-pre-processing"
Oct 30 22:07:53.184375 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184346    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fb6198-7dfd-44f0-a674-0d9287320862" containerName="main"
Oct 30 22:07:53.184375 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.184360    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd3d315-e31c-4c1f-a441-465b68dcd02f" containerName="maas-api"
Oct 30 22:07:53.188907 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.188888    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.210298 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.210267    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"]
Oct 30 22:07:53.214548 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.214508    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-589ebc-serving-cert\""
Oct 30 22:07:53.244584 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.244552    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdvf\" (UniqueName: \"kubernetes.io/projected/4ccd3396-e2ef-4338-aca4-3d3fe3296910-kube-api-access-xsdvf\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.244768 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.244611    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccd3396-e2ef-4338-aca4-3d3fe3296910-openshift-service-ca\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.244859 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.244762    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-maas-api-tls\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.244859 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.244824    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-metrics-certs\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.346617 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.346571    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdvf\" (UniqueName: \"kubernetes.io/projected/4ccd3396-e2ef-4338-aca4-3d3fe3296910-kube-api-access-xsdvf\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.346807 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.346728    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccd3396-e2ef-4338-aca4-3d3fe3296910-openshift-service-ca\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.346881 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.346851    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-maas-api-tls\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.346942 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.346908    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-metrics-certs\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.347479 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.347451    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccd3396-e2ef-4338-aca4-3d3fe3296910-openshift-service-ca\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.349487 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.349463    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-maas-api-tls\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.349581 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.349507    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-metrics-certs\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.365809 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.365786    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdvf\" (UniqueName: \"kubernetes.io/projected/4ccd3396-e2ef-4338-aca4-3d3fe3296910-kube-api-access-xsdvf\") pod \"maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.506961 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.506875    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:53.819505 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:53.819478    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"]
Oct 30 22:07:53.822117 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:07:53.822082    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccd3396_e2ef_4338_aca4_3d3fe3296910.slice/crio-a3aa8f2a0c8583396b76c8d27eedfb1dbfcafb0fa5009a9d1986d10ec5f8fea6 WatchSource:0}: Error finding container a3aa8f2a0c8583396b76c8d27eedfb1dbfcafb0fa5009a9d1986d10ec5f8fea6: Status 404 returned error can't find the container with id a3aa8f2a0c8583396b76c8d27eedfb1dbfcafb0fa5009a9d1986d10ec5f8fea6
Oct 30 22:07:54.781565 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:54.781523    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q" event={"ID":"4ccd3396-e2ef-4338-aca4-3d3fe3296910","Type":"ContainerStarted","Data":"c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc"}
Oct 30 22:07:54.781953 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:54.781585    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q" event={"ID":"4ccd3396-e2ef-4338-aca4-3d3fe3296910","Type":"ContainerStarted","Data":"a3aa8f2a0c8583396b76c8d27eedfb1dbfcafb0fa5009a9d1986d10ec5f8fea6"}
Oct 30 22:07:54.781953 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:54.781615    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:07:54.820579 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:54.820524    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q" podStartSLOduration=1.413472495 podStartE2EDuration="1.820510493s" totalImagesPullingTime="407.037998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:54.819364165 +0000 UTC m=+3474.948615623" watchObservedRunningTime="2026-10-30 22:07:54.820510493 +0000 UTC m=+3474.949761960"
Oct 30 22:07:56.994378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:07:56.994341    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"]
Oct 30 22:08:00.795825 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:00.795794    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:08:01.995074 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:01.995024    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj" podUID="662816b4-6b02-436a-92b8-000cd7ae75c0" containerName="payload-processing" containerID="cri-o://a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90" gracePeriod=25
Oct 30 22:08:02.332099 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.332070    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:08:02.440203 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.440149    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-routing-overlay-volume\") pod \"662816b4-6b02-436a-92b8-000cd7ae75c0\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") "
Oct 30 22:08:02.440380 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.440322    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-plugins-config-volume\") pod \"662816b4-6b02-436a-92b8-000cd7ae75c0\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") "
Oct 30 22:08:02.440380 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.440376    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-kube-api-access-h9q6j\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-kube-api-access-h9q6j\") pod \"662816b4-6b02-436a-92b8-000cd7ae75c0\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") "
Oct 30 22:08:02.440513 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.440480    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-routing-overlay-volume" pod "662816b4-6b02-436a-92b8-000cd7ae75c0" (UID: "662816b4-6b02-436a-92b8-000cd7ae75c0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:02.440575 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.440533    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-provider-credentials\") pod \"662816b4-6b02-436a-92b8-000cd7ae75c0\" (UID: \"662816b4-6b02-436a-92b8-000cd7ae75c0\") "
Oct 30 22:08:02.440764 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.440740    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-plugins-config-volume" pod "662816b4-6b02-436a-92b8-000cd7ae75c0" (UID: "662816b4-6b02-436a-92b8-000cd7ae75c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:02.441173 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.441141    5886 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-routing-overlay-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:02.441241 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.441175    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/662816b4-6b02-436a-92b8-000cd7ae75c0-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:02.442723 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.442696    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-provider-credentials" pod "662816b4-6b02-436a-92b8-000cd7ae75c0" (UID: "662816b4-6b02-436a-92b8-000cd7ae75c0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.442831 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.442727    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-kube-api-access-h9q6j" pod "662816b4-6b02-436a-92b8-000cd7ae75c0" (UID: "662816b4-6b02-436a-92b8-000cd7ae75c0"). InnerVolumeSpecName "kube-api-access-h9q6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:02.542749 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.542720    5886 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-provider-credentials\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:02.542749 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.542744    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9q6j\" (UniqueName: \"kubernetes.io/projected/662816b4-6b02-436a-92b8-000cd7ae75c0-kube-api-access-h9q6j\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:02.957950 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.957908    5886 generic.go:356] "Generic (PLEG): container finished" podID="662816b4-6b02-436a-92b8-000cd7ae75c0" containerID="a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90" exitCode=0
Oct 30 22:08:02.958135 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.957972    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"
Oct 30 22:08:02.958135 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.957976    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj" event={"ID":"662816b4-6b02-436a-92b8-000cd7ae75c0","Type":"ContainerDied","Data":"a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90"}
Oct 30 22:08:02.958135 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.958059    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj" event={"ID":"662816b4-6b02-436a-92b8-000cd7ae75c0","Type":"ContainerDied","Data":"464bcfef91f2d93df84cad42beca0b812e892aeacc329e7a65fc6eda3217ee18"}
Oct 30 22:08:02.958135 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.958082    5886 scope.go:122] "RemoveContainer" containerID="a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90"
Oct 30 22:08:02.978012 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.977969    5886 scope.go:122] "RemoveContainer" containerID="a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90"
Oct 30 22:08:02.980794 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:08:02.980748    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90\": container with ID starting with a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90 not found: ID does not exist" containerID="a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90"
Oct 30 22:08:02.980928 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:02.980799    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90"} err="failed to get container status \"a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90\": rpc error: code = NotFound desc = could not find container \"a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90\": container with ID starting with a8e7fc0896934e040210d79960a7bd6a6be40dff54c35d117c61f618f6a96c90 not found: ID does not exist"
Oct 30 22:08:03.008107 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:03.008079    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"]
Oct 30 22:08:03.019263 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:03.019241    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3d5d13/payload-processing-external-model-e2e-extmod-a-3d5d13-5f859nccj"]
Oct 30 22:08:04.003100 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:04.003063    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662816b4-6b02-436a-92b8-000cd7ae75c0" path="/var/lib/kubelet/pods/662816b4-6b02-436a-92b8-000cd7ae75c0/volumes"
Oct 30 22:08:11.061844 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:08:11.061808    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:08:23.873192 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:23.865723    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"]
Oct 30 22:08:23.873192 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:23.866041    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy" containerID="cri-o://bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110" gracePeriod=30
Oct 30 22:08:29.206038 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.205680    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 22:08:29.246042 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.246000    5886 generic.go:356] "Generic (PLEG): container finished" podID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerID="bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110" exitCode=0
Oct 30 22:08:29.246251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.246055    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"
Oct 30 22:08:29.246251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.246054    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" event={"ID":"ffbca04c-a3b0-4159-976c-dfa3683e1595","Type":"ContainerDied","Data":"bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110"}
Oct 30 22:08:29.246251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.246209    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k" event={"ID":"ffbca04c-a3b0-4159-976c-dfa3683e1595","Type":"ContainerDied","Data":"bd8b53f02018836b2b09874450fd9f9aff3d3580caf672c9a41009f389504ef1"}
Oct 30 22:08:29.246251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.246243    5886 scope.go:122] "RemoveContainer" containerID="bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110"
Oct 30 22:08:29.257129 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.257110    5886 scope.go:122] "RemoveContainer" containerID="bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110"
Oct 30 22:08:29.257466 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:08:29.257446    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110\": container with ID starting with bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110 not found: ID does not exist" containerID="bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110"
Oct 30 22:08:29.257549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.257475    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110"} err="failed to get container status \"bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110\": rpc error: code = NotFound desc = could not find container \"bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110\": container with ID starting with bed86b0d3923ec72d1e7e420af9bbbd5a75009a604f5daebcfd1b5a7c7e97110 not found: ID does not exist"
Oct 30 22:08:29.315778 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.315739    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-data\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.316071 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.316034    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-envoy\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.316321 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.316294    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-certs\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.316548 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.316530    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffbca04c-a3b0-4159-976c-dfa3683e1595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbca04c-a3b0-4159-976c-dfa3683e1595-istiod-ca-cert\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.316694 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.316676    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-socket\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.316910 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.316893    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-token\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-token\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.317171 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.317139    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-credential-socket\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.317327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.317307    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-kube-api-access-p8m4s\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-kube-api-access-p8m4s\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.317563 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.317544    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-podinfo\") pod \"ffbca04c-a3b0-4159-976c-dfa3683e1595\" (UID: \"ffbca04c-a3b0-4159-976c-dfa3683e1595\") "
Oct 30 22:08:29.319206 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.319076    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-socket" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.319206 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.319121    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-data" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.319206 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.319193    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbca04c-a3b0-4159-976c-dfa3683e1595-istiod-ca-cert" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:29.319463 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.319309    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-certs" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.319920 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.319816    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-credential-socket" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.321904 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.321879    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-podinfo" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:29.322269 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.322141    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-kube-api-access-p8m4s" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "kube-api-access-p8m4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:29.322269 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.322243    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-envoy" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:29.323023 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.322993    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-token" pod "ffbca04c-a3b0-4159-976c-dfa3683e1595" (UID: "ffbca04c-a3b0-4159-976c-dfa3683e1595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:29.419710 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419669    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.419966 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419720    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.419966 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419737    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.419966 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419750    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffbca04c-a3b0-4159-976c-dfa3683e1595-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.419966 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419763    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.419966 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419776    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.419966 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419789    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffbca04c-a3b0-4159-976c-dfa3683e1595-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.419966 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419804    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8m4s\" (UniqueName: \"kubernetes.io/projected/ffbca04c-a3b0-4159-976c-dfa3683e1595-kube-api-access-p8m4s\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.419966 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.419819    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffbca04c-a3b0-4159-976c-dfa3683e1595-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:08:29.603674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.603633    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"]
Oct 30 22:08:29.613419 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.613388    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-49738b-openshift-default-57bbcfc767-vnj5k"]
Oct 30 22:08:29.992279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:08:29.992241    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" path="/var/lib/kubelet/pods/ffbca04c-a3b0-4159-976c-dfa3683e1595/volumes"
Oct 30 22:08:41.031449 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:08:41.031365    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:09:11.031670 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:09:11.031630    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:09:21.619824 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:21.619775    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"]
Oct 30 22:09:21.620704 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:21.620670    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy" containerID="cri-o://2f53d910a5d0c01e669fa7ef6f649cd678bcfb37cc4a6ac11d9102656fd8c7a5" gracePeriod=30
Oct 30 22:09:24.140339 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:24.140301    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:09:24.140738 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:24.140368    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:26.869987 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.869893    5886 generic.go:356] "Generic (PLEG): container finished" podID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerID="2f53d910a5d0c01e669fa7ef6f649cd678bcfb37cc4a6ac11d9102656fd8c7a5" exitCode=0
Oct 30 22:09:26.870543 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.870050    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" event={"ID":"59ef2a33-4d03-455e-b374-6b7f7224e4b1","Type":"ContainerDied","Data":"2f53d910a5d0c01e669fa7ef6f649cd678bcfb37cc4a6ac11d9102656fd8c7a5"}
Oct 30 22:09:26.924515 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.924486    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:09:26.977537 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.977498    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-socket\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.977752 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.977610    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-kube-api-access-2cz5n\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-kube-api-access-2cz5n\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.977752 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.977688    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-credential-socket\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.977752 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.977714    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istiod-ca-cert\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.977752 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.977743    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-certs\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.977978 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.977779    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-data\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.977978 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.977864    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-podinfo\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.977978 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.977914    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-envoy\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.978148 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.978022    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-token\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-token\") pod \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\" (UID: \"59ef2a33-4d03-455e-b374-6b7f7224e4b1\") "
Oct 30 22:09:26.979287 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.979195    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-credential-socket" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:26.979287 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.979222    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-socket" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:26.979287 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.979260    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-certs" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:26.979577 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.979417    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-data" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:26.979811 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.979786    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istiod-ca-cert" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:26.984975 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.984926    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-podinfo" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:26.990720 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.990684    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-kube-api-access-2cz5n" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "kube-api-access-2cz5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:26.990954 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.990924    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-token" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:26.991035 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:26.990942    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-envoy" pod "59ef2a33-4d03-455e-b374-6b7f7224e4b1" (UID: "59ef2a33-4d03-455e-b374-6b7f7224e4b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:27.039148 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.039115    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"]
Oct 30 22:09:27.044093 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.044067    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"]
Oct 30 22:09:27.081398 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081345    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.081398 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081377    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.081398 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081393    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.081398 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081405    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.085039 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081418    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.085039 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081431    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.085039 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081443    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.085039 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081455    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ef2a33-4d03-455e-b374-6b7f7224e4b1-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.085039 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.081469    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cz5n\" (UniqueName: \"kubernetes.io/projected/59ef2a33-4d03-455e-b374-6b7f7224e4b1-kube-api-access-2cz5n\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:27.881842 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.881800    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr" event={"ID":"59ef2a33-4d03-455e-b374-6b7f7224e4b1","Type":"ContainerDied","Data":"c101b9aad948aaeb85e7f167f648289ee11e6e7e9671eec7d009d9a9cf359bb6"}
Oct 30 22:09:27.882339 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.881870    5886 scope.go:122] "RemoveContainer" containerID="2f53d910a5d0c01e669fa7ef6f649cd678bcfb37cc4a6ac11d9102656fd8c7a5"
Oct 30 22:09:27.882339 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.881811    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"
Oct 30 22:09:27.947199 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.947150    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"]
Oct 30 22:09:27.952390 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.952366    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdb168-openshift-default-9b9894fc7-hlcbr"]
Oct 30 22:09:27.991864 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:27.991819    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" path="/var/lib/kubelet/pods/59ef2a33-4d03-455e-b374-6b7f7224e4b1/volumes"
Oct 30 22:09:32.766971 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:32.766932    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"]
Oct 30 22:09:32.798340 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:32.798300    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"]
Oct 30 22:09:37.767723 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:37.767669    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn" podUID="11c5825c-9673-43e1-9316-9e98058a4fa4" containerName="payload-processing" containerID="cri-o://90c04938f696adaf53c4ad0652e1062858ba661522ec42c3a6eba4553bbb1b2e" gracePeriod=25
Oct 30 22:09:37.798731 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:37.798670    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6" podUID="017cb3c5-df91-42c3-9061-e5545a46a365" containerName="payload-pre-processing" containerID="cri-o://9277bd1731c897db71eb5524324ab09a90da18b1528d015b92548c3069617d0b" gracePeriod=25
Oct 30 22:09:40.009996 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:40.009953    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"]
Oct 30 22:09:40.010610 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:40.010242    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy" containerID="cri-o://e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3" gracePeriod=30
Oct 30 22:09:41.029984 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:09:41.029931    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:09:42.064281 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.064244    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main" containerID="cri-o://b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f" gracePeriod=45
Oct 30 22:09:42.066142 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.066108    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main" containerID="cri-o://0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f" gracePeriod=45
Oct 30 22:09:42.385878 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.385530    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 22:09:42.391066 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.391037    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 22:09:42.523895 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.523860    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-home\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-home\") pod \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") "
Oct 30 22:09:42.524086 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.523917    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-model-cache\") pod \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") "
Oct 30 22:09:42.524086 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.523969    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-dshm\") pod \"1020a874-7ec4-44be-905c-792bc7f714e1\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") "
Oct 30 22:09:42.524086 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524017    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-dshm\") pod \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") "
Oct 30 22:09:42.524505 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524323    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-model-cache" pod "199b2e2a-8baa-47fd-9361-a3018b22fc4f" (UID: "199b2e2a-8baa-47fd-9361-a3018b22fc4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:42.524505 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524493    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tls-certs\") pod \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") "
Oct 30 22:09:42.524505 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524431    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-home" pod "199b2e2a-8baa-47fd-9361-a3018b22fc4f" (UID: "199b2e2a-8baa-47fd-9361-a3018b22fc4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:42.524784 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524576    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tmp-dir\") pod \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") "
Oct 30 22:09:42.524784 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524663    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1020a874-7ec4-44be-905c-792bc7f714e1-kube-api-access-zkcpr\" (UniqueName: \"kubernetes.io/projected/1020a874-7ec4-44be-905c-792bc7f714e1-kube-api-access-zkcpr\") pod \"1020a874-7ec4-44be-905c-792bc7f714e1\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") "
Oct 30 22:09:42.524784 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524774    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1020a874-7ec4-44be-905c-792bc7f714e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/1020a874-7ec4-44be-905c-792bc7f714e1-tls-certs\") pod \"1020a874-7ec4-44be-905c-792bc7f714e1\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") "
Oct 30 22:09:42.524951 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524805    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-model-cache\") pod \"1020a874-7ec4-44be-905c-792bc7f714e1\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") "
Oct 30 22:09:42.524951 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524847    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tmp-dir" pod "199b2e2a-8baa-47fd-9361-a3018b22fc4f" (UID: "199b2e2a-8baa-47fd-9361-a3018b22fc4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:42.524951 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524896    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-tmp-dir\") pod \"1020a874-7ec4-44be-905c-792bc7f714e1\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") "
Oct 30 22:09:42.525115 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524968    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-home\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-home\") pod \"1020a874-7ec4-44be-905c-792bc7f714e1\" (UID: \"1020a874-7ec4-44be-905c-792bc7f714e1\") "
Oct 30 22:09:42.525115 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.524997    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199b2e2a-8baa-47fd-9361-a3018b22fc4f-kube-api-access-5tqcj\" (UniqueName: \"kubernetes.io/projected/199b2e2a-8baa-47fd-9361-a3018b22fc4f-kube-api-access-5tqcj\") pod \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\" (UID: \"199b2e2a-8baa-47fd-9361-a3018b22fc4f\") "
Oct 30 22:09:42.525396 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.525346    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-tmp-dir" pod "1020a874-7ec4-44be-905c-792bc7f714e1" (UID: "1020a874-7ec4-44be-905c-792bc7f714e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:42.525396 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.525360    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-home" pod "1020a874-7ec4-44be-905c-792bc7f714e1" (UID: "1020a874-7ec4-44be-905c-792bc7f714e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:42.525619 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.525396    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-model-cache" pod "1020a874-7ec4-44be-905c-792bc7f714e1" (UID: "1020a874-7ec4-44be-905c-792bc7f714e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:42.526087 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.526043    5886 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.526087 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.526070    5886 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.526087 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.526083    5886 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.526360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.526097    5886 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.526360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.526110    5886 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.526360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.526122    5886 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.526736 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.526710    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-dshm" pod "1020a874-7ec4-44be-905c-792bc7f714e1" (UID: "1020a874-7ec4-44be-905c-792bc7f714e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:42.527138 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.527110    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-dshm" pod "199b2e2a-8baa-47fd-9361-a3018b22fc4f" (UID: "199b2e2a-8baa-47fd-9361-a3018b22fc4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:42.527267 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.527151    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tls-certs" pod "199b2e2a-8baa-47fd-9361-a3018b22fc4f" (UID: "199b2e2a-8baa-47fd-9361-a3018b22fc4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:42.527317 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.527278    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1020a874-7ec4-44be-905c-792bc7f714e1-tls-certs" pod "1020a874-7ec4-44be-905c-792bc7f714e1" (UID: "1020a874-7ec4-44be-905c-792bc7f714e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:42.527446 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.527422    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1020a874-7ec4-44be-905c-792bc7f714e1-kube-api-access-zkcpr" pod "1020a874-7ec4-44be-905c-792bc7f714e1" (UID: "1020a874-7ec4-44be-905c-792bc7f714e1"). InnerVolumeSpecName "kube-api-access-zkcpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:42.528426 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.528399    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199b2e2a-8baa-47fd-9361-a3018b22fc4f-kube-api-access-5tqcj" pod "199b2e2a-8baa-47fd-9361-a3018b22fc4f" (UID: "199b2e2a-8baa-47fd-9361-a3018b22fc4f"). InnerVolumeSpecName "kube-api-access-5tqcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:42.597442 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.597351    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6" podUID="017cb3c5-df91-42c3-9061-e5545a46a365" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:42.597618 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.597467    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn" podUID="11c5825c-9673-43e1-9316-9e98058a4fa4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:42.627490 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.627462    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkcpr\" (UniqueName: \"kubernetes.io/projected/1020a874-7ec4-44be-905c-792bc7f714e1-kube-api-access-zkcpr\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.627490 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.627488    5886 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1020a874-7ec4-44be-905c-792bc7f714e1-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.627673 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.627502    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tqcj\" (UniqueName: \"kubernetes.io/projected/199b2e2a-8baa-47fd-9361-a3018b22fc4f-kube-api-access-5tqcj\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.627673 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.627511    5886 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1020a874-7ec4-44be-905c-792bc7f714e1-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.627673 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.627518    5886 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199b2e2a-8baa-47fd-9361-a3018b22fc4f-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:42.627673 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:42.627526    5886 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2e2a-8baa-47fd-9361-a3018b22fc4f-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:43.029797 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.029757    5886 generic.go:356] "Generic (PLEG): container finished" podID="1020a874-7ec4-44be-905c-792bc7f714e1" containerID="0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f" exitCode=0
Oct 30 22:09:43.030005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.029816    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"
Oct 30 22:09:43.030005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.029837    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" event={"ID":"1020a874-7ec4-44be-905c-792bc7f714e1","Type":"ContainerDied","Data":"0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f"}
Oct 30 22:09:43.030005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.029881    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz" event={"ID":"1020a874-7ec4-44be-905c-792bc7f714e1","Type":"ContainerDied","Data":"681fcb69096aa040e27b70711494f6ae7b96c6951383f715ae7fac026b2d4488"}
Oct 30 22:09:43.030005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.029900    5886 scope.go:122] "RemoveContainer" containerID="0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f"
Oct 30 22:09:43.031648 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.031621    5886 generic.go:356] "Generic (PLEG): container finished" podID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerID="b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f" exitCode=0
Oct 30 22:09:43.031788 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.031657    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"
Oct 30 22:09:43.031788 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.031653    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" event={"ID":"199b2e2a-8baa-47fd-9361-a3018b22fc4f","Type":"ContainerDied","Data":"b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f"}
Oct 30 22:09:43.031788 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.031770    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk" event={"ID":"199b2e2a-8baa-47fd-9361-a3018b22fc4f","Type":"ContainerDied","Data":"5aaf85b3a6b90aea78cd35914db65fd11a0a98980924e5f796341cf616252b9b"}
Oct 30 22:09:43.041868 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.041833    5886 scope.go:122] "RemoveContainer" containerID="0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f"
Oct 30 22:09:43.042226 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:09:43.042141    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f\": container with ID starting with 0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f not found: ID does not exist" containerID="0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f"
Oct 30 22:09:43.042226 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.042209    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f"} err="failed to get container status \"0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f\": rpc error: code = NotFound desc = could not find container \"0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f\": container with ID starting with 0afb6700465b45f31ea9af9471b421015d957806a496473dcb04c74464546d2f not found: ID does not exist"
Oct 30 22:09:43.042389 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.042234    5886 scope.go:122] "RemoveContainer" containerID="b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f"
Oct 30 22:09:43.053536 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.053516    5886 scope.go:122] "RemoveContainer" containerID="b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f"
Oct 30 22:09:43.053854 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:09:43.053828    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f\": container with ID starting with b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f not found: ID does not exist" containerID="b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f"
Oct 30 22:09:43.053957 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.053859    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f"} err="failed to get container status \"b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f\": rpc error: code = NotFound desc = could not find container \"b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f\": container with ID starting with b4dbb834968e6b2d848c7c6bbc2b4b34f2dd99fd1ce0ca2e0b0cd5c4d61d6c2f not found: ID does not exist"
Oct 30 22:09:43.081773 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.081746    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"]
Oct 30 22:09:43.085911 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.085884    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-w3-50ab77-kserve-5dd6bcf454-gqvrz"]
Oct 30 22:09:43.117509 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.117466    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"]
Oct 30 22:09:43.122053 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.122029    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-50ab77/distinct-2-w3-50ab77-kserve-8c6dccd5c-7tctk"]
Oct 30 22:09:43.993285 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.993230    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" path="/var/lib/kubelet/pods/1020a874-7ec4-44be-905c-792bc7f714e1/volumes"
Oct 30 22:09:43.993934 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:43.993908    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" path="/var/lib/kubelet/pods/199b2e2a-8baa-47fd-9361-a3018b22fc4f/volumes"
Oct 30 22:09:45.296972 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.296944    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:09:45.368981 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.368928    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-kube-api-access-sf6gj\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-kube-api-access-sf6gj\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.369251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.369060    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-envoy\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.369251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.369122    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-data\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.369251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.369173    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-token\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-token\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.369251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.369213    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6054e7ae-175c-460b-8de7-6c27cbf501a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6054e7ae-175c-460b-8de7-6c27cbf501a3-istiod-ca-cert\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.369588 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.369254    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-certs\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.369588 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.369287    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-podinfo\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.369588 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.369317    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-credential-socket\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.369588 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.369372    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-socket\") pod \"6054e7ae-175c-460b-8de7-6c27cbf501a3\" (UID: \"6054e7ae-175c-460b-8de7-6c27cbf501a3\") "
Oct 30 22:09:45.370101 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.370054    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6054e7ae-175c-460b-8de7-6c27cbf501a3-istiod-ca-cert" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:45.370382 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.370343    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-socket" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.370753 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.370549    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-certs" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.370858 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.370769    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-data" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.370858 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.370784    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-credential-socket" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.373845 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.373805    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-envoy" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:45.375902 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.375863    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-token" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:45.376016 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.375941    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-kube-api-access-sf6gj" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "kube-api-access-sf6gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:45.376595 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.376571    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-podinfo" pod "6054e7ae-175c-460b-8de7-6c27cbf501a3" (UID: "6054e7ae-175c-460b-8de7-6c27cbf501a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:45.471860 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471822    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:45.471860 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471856    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:45.471860 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471865    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6054e7ae-175c-460b-8de7-6c27cbf501a3-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:45.471860 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471874    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:45.472283 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471882    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:45.472283 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471895    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:45.472283 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471912    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:45.472283 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471924    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf6gj\" (UniqueName: \"kubernetes.io/projected/6054e7ae-175c-460b-8de7-6c27cbf501a3-kube-api-access-sf6gj\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:45.472283 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:45.471934    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6054e7ae-175c-460b-8de7-6c27cbf501a3-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:46.061922 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.061875    5886 generic.go:356] "Generic (PLEG): container finished" podID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerID="e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3" exitCode=0
Oct 30 22:09:46.062115 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.061940    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"
Oct 30 22:09:46.062115 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.061959    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" event={"ID":"6054e7ae-175c-460b-8de7-6c27cbf501a3","Type":"ContainerDied","Data":"e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3"}
Oct 30 22:09:46.062115 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.062030    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b" event={"ID":"6054e7ae-175c-460b-8de7-6c27cbf501a3","Type":"ContainerDied","Data":"374cbf2e2c502b88b002ab751d37faa60dedb8604cf82da179d15c5f78bdbcae"}
Oct 30 22:09:46.062115 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.062073    5886 scope.go:122] "RemoveContainer" containerID="e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3"
Oct 30 22:09:46.073315 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.073288    5886 scope.go:122] "RemoveContainer" containerID="e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3"
Oct 30 22:09:46.073700 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:09:46.073667    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3\": container with ID starting with e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3 not found: ID does not exist" containerID="e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3"
Oct 30 22:09:46.073815 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.073715    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3"} err="failed to get container status \"e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3\": rpc error: code = NotFound desc = could not find container \"e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3\": container with ID starting with e73248a309839476adeaf68925f9d063299628ec2913171a6a42a7caa1ecc6c3 not found: ID does not exist"
Oct 30 22:09:46.116761 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.116727    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"]
Oct 30 22:09:46.151415 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:46.151371    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-846656-openshift-default-6447554d6f-ncr7b"]
Oct 30 22:09:47.989405 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:47.989373    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" path="/var/lib/kubelet/pods/6054e7ae-175c-460b-8de7-6c27cbf501a3/volumes"
Oct 30 22:09:52.598226 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:52.598145    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6" podUID="017cb3c5-df91-42c3-9061-e5545a46a365" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:52.598808 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:52.598635    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn" podUID="11c5825c-9673-43e1-9316-9e98058a4fa4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:09:58.183553 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.183509    5886 generic.go:356] "Generic (PLEG): container finished" podID="11c5825c-9673-43e1-9316-9e98058a4fa4" containerID="90c04938f696adaf53c4ad0652e1062858ba661522ec42c3a6eba4553bbb1b2e" exitCode=0
Oct 30 22:09:58.183998 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.183611    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn" event={"ID":"11c5825c-9673-43e1-9316-9e98058a4fa4","Type":"ContainerDied","Data":"90c04938f696adaf53c4ad0652e1062858ba661522ec42c3a6eba4553bbb1b2e"}
Oct 30 22:09:58.183998 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.183664    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn" event={"ID":"11c5825c-9673-43e1-9316-9e98058a4fa4","Type":"ContainerDied","Data":"6e5f6e39a7ac018ad4be1a8da58bec26a90a225f1cb0aa2d0b7c7ed5373c34cb"}
Oct 30 22:09:58.183998 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.183686    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e5f6e39a7ac018ad4be1a8da58bec26a90a225f1cb0aa2d0b7c7ed5373c34cb"
Oct 30 22:09:58.186353 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.186326    5886 generic.go:356] "Generic (PLEG): container finished" podID="017cb3c5-df91-42c3-9061-e5545a46a365" containerID="9277bd1731c897db71eb5524324ab09a90da18b1528d015b92548c3069617d0b" exitCode=0
Oct 30 22:09:58.186574 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.186502    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6" event={"ID":"017cb3c5-df91-42c3-9061-e5545a46a365","Type":"ContainerDied","Data":"9277bd1731c897db71eb5524324ab09a90da18b1528d015b92548c3069617d0b"}
Oct 30 22:09:58.191838 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.191816    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 22:09:58.214150 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.214130    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 22:09:58.264776 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.264738    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c5825c-9673-43e1-9316-9e98058a4fa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c5825c-9673-43e1-9316-9e98058a4fa4-plugins-config-volume\") pod \"11c5825c-9673-43e1-9316-9e98058a4fa4\" (UID: \"11c5825c-9673-43e1-9316-9e98058a4fa4\") "
Oct 30 22:09:58.265031 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.264977    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c5825c-9673-43e1-9316-9e98058a4fa4-kube-api-access-6ppxn\" (UniqueName: \"kubernetes.io/projected/11c5825c-9673-43e1-9316-9e98058a4fa4-kube-api-access-6ppxn\") pod \"11c5825c-9673-43e1-9316-9e98058a4fa4\" (UID: \"11c5825c-9673-43e1-9316-9e98058a4fa4\") "
Oct 30 22:09:58.265626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.265350    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c5825c-9673-43e1-9316-9e98058a4fa4-plugins-config-volume" pod "11c5825c-9673-43e1-9316-9e98058a4fa4" (UID: "11c5825c-9673-43e1-9316-9e98058a4fa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:58.265626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.265592    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c5825c-9673-43e1-9316-9e98058a4fa4-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:58.267968 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.267938    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c5825c-9673-43e1-9316-9e98058a4fa4-kube-api-access-6ppxn" pod "11c5825c-9673-43e1-9316-9e98058a4fa4" (UID: "11c5825c-9673-43e1-9316-9e98058a4fa4"). InnerVolumeSpecName "kube-api-access-6ppxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:58.367002 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.366908    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017cb3c5-df91-42c3-9061-e5545a46a365-kube-api-access-rn68b\" (UniqueName: \"kubernetes.io/projected/017cb3c5-df91-42c3-9061-e5545a46a365-kube-api-access-rn68b\") pod \"017cb3c5-df91-42c3-9061-e5545a46a365\" (UID: \"017cb3c5-df91-42c3-9061-e5545a46a365\") "
Oct 30 22:09:58.367002 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.366964    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017cb3c5-df91-42c3-9061-e5545a46a365-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017cb3c5-df91-42c3-9061-e5545a46a365-plugins-config-volume\") pod \"017cb3c5-df91-42c3-9061-e5545a46a365\" (UID: \"017cb3c5-df91-42c3-9061-e5545a46a365\") "
Oct 30 22:09:58.367350 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.367320    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017cb3c5-df91-42c3-9061-e5545a46a365-plugins-config-volume" pod "017cb3c5-df91-42c3-9061-e5545a46a365" (UID: "017cb3c5-df91-42c3-9061-e5545a46a365"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:58.367568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.367541    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ppxn\" (UniqueName: \"kubernetes.io/projected/11c5825c-9673-43e1-9316-9e98058a4fa4-kube-api-access-6ppxn\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:58.367568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.367560    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017cb3c5-df91-42c3-9061-e5545a46a365-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:58.369122 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.369098    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017cb3c5-df91-42c3-9061-e5545a46a365-kube-api-access-rn68b" pod "017cb3c5-df91-42c3-9061-e5545a46a365" (UID: "017cb3c5-df91-42c3-9061-e5545a46a365"). InnerVolumeSpecName "kube-api-access-rn68b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:58.469356 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:58.469322    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn68b\" (UniqueName: \"kubernetes.io/projected/017cb3c5-df91-42c3-9061-e5545a46a365-kube-api-access-rn68b\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:09:59.195199 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.195129    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6" event={"ID":"017cb3c5-df91-42c3-9061-e5545a46a365","Type":"ContainerDied","Data":"6d6b90ebe6cdd82c94570de9236f8d63558648c62e685008003db13027c2d8af"}
Oct 30 22:09:59.195199 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.195186    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"
Oct 30 22:09:59.195753 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.195224    5886 scope.go:122] "RemoveContainer" containerID="9277bd1731c897db71eb5524324ab09a90da18b1528d015b92548c3069617d0b"
Oct 30 22:09:59.195753 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.195171    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"
Oct 30 22:09:59.261500 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.261460    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"]
Oct 30 22:09:59.273013 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.272982    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-50ab77-bc4897bdc-p72f6"]
Oct 30 22:09:59.298396 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.298363    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"]
Oct 30 22:09:59.302529 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.302501    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-50ab77-8649598749-rmbnn"]
Oct 30 22:09:59.989751 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.989715    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017cb3c5-df91-42c3-9061-e5545a46a365" path="/var/lib/kubelet/pods/017cb3c5-df91-42c3-9061-e5545a46a365/volumes"
Oct 30 22:09:59.990199 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:09:59.990178    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c5825c-9673-43e1-9316-9e98058a4fa4" path="/var/lib/kubelet/pods/11c5825c-9673-43e1-9316-9e98058a4fa4/volumes"
Oct 30 22:10:02.700278 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.700240    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"]
Oct 30 22:10:02.700934 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.700918    5886 cpu_manager.go:414] "containerMap: removing container" podUID="017cb3c5-df91-42c3-9061-e5545a46a365" containerName="payload-pre-processing"
Oct 30 22:10:02.700986 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.700935    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017cb3c5-df91-42c3-9061-e5545a46a365" containerName="payload-pre-processing"
Oct 30 22:10:02.700986 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.700968    5886 cpu_manager.go:414] "containerMap: removing container" podUID="662816b4-6b02-436a-92b8-000cd7ae75c0" containerName="payload-processing"
Oct 30 22:10:02.700986 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.700974    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662816b4-6b02-436a-92b8-000cd7ae75c0" containerName="payload-processing"
Oct 30 22:10:02.700986 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.700985    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.700990    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701002    5886 cpu_manager.go:414] "containerMap: removing container" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701006    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701016    5886 cpu_manager.go:414] "containerMap: removing container" podUID="11c5825c-9673-43e1-9316-9e98058a4fa4" containerName="payload-processing"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701021    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c5825c-9673-43e1-9316-9e98058a4fa4" containerName="payload-processing"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701036    5886 cpu_manager.go:414] "containerMap: removing container" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701040    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701084    5886 cpu_manager.go:414] "containerMap: removing container" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701090    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701097    5886 cpu_manager.go:414] "containerMap: removing container" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy"
Oct 30 22:10:02.701141 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701102    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy"
Oct 30 22:10:02.701685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701286    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="662816b4-6b02-436a-92b8-000cd7ae75c0" containerName="payload-processing"
Oct 30 22:10:02.701685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701300    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffbca04c-a3b0-4159-976c-dfa3683e1595" containerName="istio-proxy"
Oct 30 22:10:02.701685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701315    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="1020a874-7ec4-44be-905c-792bc7f714e1" containerName="main"
Oct 30 22:10:02.701685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701327    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c5825c-9673-43e1-9316-9e98058a4fa4" containerName="payload-processing"
Oct 30 22:10:02.701685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701337    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="6054e7ae-175c-460b-8de7-6c27cbf501a3" containerName="istio-proxy"
Oct 30 22:10:02.701685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701344    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="199b2e2a-8baa-47fd-9361-a3018b22fc4f" containerName="main"
Oct 30 22:10:02.701685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701350    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ef2a33-4d03-455e-b374-6b7f7224e4b1" containerName="istio-proxy"
Oct 30 22:10:02.701685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.701357    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="017cb3c5-df91-42c3-9061-e5545a46a365" containerName="payload-pre-processing"
Oct 30 22:10:02.706623 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.706600    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.733634 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.733608    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-50ab77-openshift-default-dockercfg-tt79b\""
Oct 30 22:10:02.737581 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.737555    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"]
Oct 30 22:10:02.835564 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.835522    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb6383b-3962-43f7-be89-29de959289ea-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.835744 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.835595    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2p4\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-kube-api-access-xr2p4\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.835744 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.835715    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.835820 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.835754    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.835929 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.835901    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.835970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.835950    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb6383b-3962-43f7-be89-29de959289ea-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.836013 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.836000    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.836057 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.836037    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.836117 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.836073    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937008 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.936973    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2p4\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-kube-api-access-xr2p4\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937275 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937088    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937275 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937136    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937275 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937250    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937498 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937282    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb6383b-3962-43f7-be89-29de959289ea-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937498 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937334    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937498 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937380    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937498 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937434    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937704 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937518    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb6383b-3962-43f7-be89-29de959289ea-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.937874 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937840    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-certs\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.938025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937873    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-data\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.938025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937941    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.938025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.937977    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-credential-socket\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.938338 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.938318    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb6383b-3962-43f7-be89-29de959289ea-istiod-ca-cert\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.939836 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.939807    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-envoy\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.940278 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.940250    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb6383b-3962-43f7-be89-29de959289ea-istio-podinfo\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.957468 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.957401    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-istio-token\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:02.957600 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:02.957578    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2p4\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-kube-api-access-xr2p4\") pod \"e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") " pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:03.017906 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:03.017866    5886 scope.go:122] "RemoveContainer" containerID="90c04938f696adaf53c4ad0652e1062858ba661522ec42c3a6eba4553bbb1b2e"
Oct 30 22:10:03.020273 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:03.020238    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:03.364181 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:03.363982    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"]
Oct 30 22:10:03.368671 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:10:03.368633    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb6383b_3962_43f7_be89_29de959289ea.slice/crio-68203ea36e264ee9963c5735f38775e04f22118cea89e875e8929cc166c8fdc8 WatchSource:0}: Error finding container 68203ea36e264ee9963c5735f38775e04f22118cea89e875e8929cc166c8fdc8: Status 404 returned error can't find the container with id 68203ea36e264ee9963c5735f38775e04f22118cea89e875e8929cc166c8fdc8
Oct 30 22:10:03.371939 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:03.371904    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:03.372060 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:03.371972    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:03.372060 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:03.372018    5886 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 22:10:04.236870 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:04.236831    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" event={"ID":"cfb6383b-3962-43f7-be89-29de959289ea","Type":"ContainerStarted","Data":"365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4"}
Oct 30 22:10:04.237307 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:04.236898    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" event={"ID":"cfb6383b-3962-43f7-be89-29de959289ea","Type":"ContainerStarted","Data":"68203ea36e264ee9963c5735f38775e04f22118cea89e875e8929cc166c8fdc8"}
Oct 30 22:10:04.303184 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:04.301680    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podStartSLOduration=2.301656803 podStartE2EDuration="2.301656803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:04.297940181 +0000 UTC m=+3604.427191640" watchObservedRunningTime="2026-10-30 22:10:04.301656803 +0000 UTC m=+3604.430908306"
Oct 30 22:10:05.021287 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:05.021251    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:05.023199 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:05.023173    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:10:05.023280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:05.023226    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:10:06.021495 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:06.021457    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:10:06.021909 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:06.021530    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:10:07.021367 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:07.021332    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:10:07.021543 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:07.021385    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:10:08.021350 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:08.021316    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:10:08.021541 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:08.021378    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:10:09.028179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:09.023193    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:10:09.028179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:09.023265    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:10:10.020754 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:10.020719    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:10:10.020946 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:10.020774    5886 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 22:10:11.027609 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:11.027526    5886 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:11.028118 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:11.028089    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:11.028984 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:11.028960    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:11.031678 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:10:11.031645    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:10:41.028975 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:10:41.028939    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:10:44.450841 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:44.438539    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"]
Oct 30 22:10:44.450841 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:44.438802    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk" podUID="01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" containerName="maas-api" containerID="cri-o://b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697" gracePeriod=30
Oct 30 22:10:45.233688 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.233656    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 22:10:45.329980 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.329946    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls\") pod \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") "
Oct 30 22:10:45.330185 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.330007    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-metrics-certs\") pod \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") "
Oct 30 22:10:45.330185 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.330049    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-kube-api-access-w6xzm\" (UniqueName: \"kubernetes.io/projected/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-kube-api-access-w6xzm\") pod \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") "
Oct 30 22:10:45.330316 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.330204    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-openshift-service-ca\") pod \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\" (UID: \"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e\") "
Oct 30 22:10:45.331266 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.331232    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-openshift-service-ca" pod "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" (UID: "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:45.332759 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.332726    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls" pod "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" (UID: "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:45.333600 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.333576    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-metrics-certs" pod "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" (UID: "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:45.334304 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.334276    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-kube-api-access-w6xzm" pod "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" (UID: "01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e"). InnerVolumeSpecName "kube-api-access-w6xzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:45.431928 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.431892    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:45.431928 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.431927    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:45.432195 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.431941    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:45.432195 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.431954    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6xzm\" (UniqueName: \"kubernetes.io/projected/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e-kube-api-access-w6xzm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:45.715169 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.715121    5886 generic.go:356] "Generic (PLEG): container finished" podID="01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" containerID="b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697" exitCode=0
Oct 30 22:10:45.715608 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.715180    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"
Oct 30 22:10:45.715608 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.715211    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk" event={"ID":"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e","Type":"ContainerDied","Data":"b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697"}
Oct 30 22:10:45.715608 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.715268    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk" event={"ID":"01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e","Type":"ContainerDied","Data":"e58cefebd34f48e5827d17d043ad5519251fb841f77db41eb7ed9abee5128e98"}
Oct 30 22:10:45.715608 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.715298    5886 scope.go:122] "RemoveContainer" containerID="b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697"
Oct 30 22:10:45.727252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.727233    5886 scope.go:122] "RemoveContainer" containerID="b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697"
Oct 30 22:10:45.727526 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:10:45.727507    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697\": container with ID starting with b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697 not found: ID does not exist" containerID="b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697"
Oct 30 22:10:45.727567 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.727535    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697"} err="failed to get container status \"b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697\": rpc error: code = NotFound desc = could not find container \"b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697\": container with ID starting with b9cf4139e66fa721d193cd53e96b080e8004f62c6af415d7bdb9306cc3d2a697 not found: ID does not exist"
Oct 30 22:10:45.772754 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.772718    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"]
Oct 30 22:10:45.784655 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.784623    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-50ab77-6d56bc6c96-ck4dk"]
Oct 30 22:10:45.990652 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:45.990567    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" path="/var/lib/kubelet/pods/01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e/volumes"
Oct 30 22:10:52.950670 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:52.950629    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"]
Oct 30 22:10:52.951183 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:52.950953    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy" containerID="cri-o://365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4" gracePeriod=30
Oct 30 22:10:52.962228 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:52.962198    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"]
Oct 30 22:10:52.962465 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:52.962417    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" containerID="cri-o://722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8" gracePeriod=30
Oct 30 22:10:56.031212 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:56.031147    5886 patch_prober.go:29] interesting pod/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:56.031909 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:56.031231    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:57.481261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:57.481216    5886 patch_prober.go:29] interesting pod/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:57.481904 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:57.481289    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:58.250770 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.250711    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:58.286632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.286601    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 22:10:58.320568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.320529    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-data\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.320724 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.320595    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-credential-socket\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.320724 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.320629    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-certs\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.320858 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.320827    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-istio-token\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-istio-token\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.320925 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.320872    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-kube-api-access-xr2p4\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-kube-api-access-xr2p4\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.320988 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.320921    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-data" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.320988 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.320929    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb6383b-3962-43f7-be89-29de959289ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb6383b-3962-43f7-be89-29de959289ea-istiod-ca-cert\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.321108 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.321055    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-socket\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.321108 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.321093    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfb6383b-3962-43f7-be89-29de959289ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb6383b-3962-43f7-be89-29de959289ea-istio-podinfo\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.321265 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.321201    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-envoy\") pod \"cfb6383b-3962-43f7-be89-29de959289ea\" (UID: \"cfb6383b-3962-43f7-be89-29de959289ea\") "
Oct 30 22:10:58.321265 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.321247    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb6383b-3962-43f7-be89-29de959289ea-istiod-ca-cert" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:58.321821 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.321788    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-socket" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.321982 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.321823    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb6383b-3962-43f7-be89-29de959289ea-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.321982 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.321845    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.321982 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.321864    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-certs" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.324455 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.324432    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-credential-socket" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.325958 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.325615    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-kube-api-access-xr2p4" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "kube-api-access-xr2p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:58.326088 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.326003    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-istio-token" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:58.326088 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.326007    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-envoy" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.326285 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.326092    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfb6383b-3962-43f7-be89-29de959289ea-istio-podinfo" pod "cfb6383b-3962-43f7-be89-29de959289ea" (UID: "cfb6383b-3962-43f7-be89-29de959289ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:58.424257 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.423639    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-envoy\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.424257 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.424190    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-podinfo\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.424532 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.424292    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-certs\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.424532 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.424349    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-data\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.424532 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.424416    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-token\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-token\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.424532 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.424446    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-socket\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.424829 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.424600    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-credential-socket\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.424829 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.424652    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istiod-ca-cert\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.424829 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.424684    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-kube-api-access-lhkjn\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-kube-api-access-lhkjn\") pod \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\" (UID: \"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e\") "
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.432793    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-podinfo" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.433313    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-certs" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.433489    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-data" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.433792    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istiod-ca-cert" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434025    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-socket" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434768    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434795    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434812    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2p4\" (UniqueName: \"kubernetes.io/projected/cfb6383b-3962-43f7-be89-29de959289ea-kube-api-access-xr2p4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434827    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434848    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434861    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434874    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434890    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb6383b-3962-43f7-be89-29de959289ea-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434910    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434925    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434938    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.448568 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.434952    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb6383b-3962-43f7-be89-29de959289ea-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.450001 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.449937    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-credential-socket" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.455593 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.455559    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-token" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:58.455961 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.455760    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-envoy" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:58.457407 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.457377    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-kube-api-access-lhkjn" pod "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" (UID: "e5962f18-d7e2-4dc4-83c9-a1f3d784b59e"). InnerVolumeSpecName "kube-api-access-lhkjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:58.536646 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.536563    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.536646 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.536595    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.536646 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.536605    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.536646 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.536616    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkjn\" (UniqueName: \"kubernetes.io/projected/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e-kube-api-access-lhkjn\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:10:58.814762 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.814661    5886 generic.go:356] "Generic (PLEG): container finished" podID="cfb6383b-3962-43f7-be89-29de959289ea" containerID="365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4" exitCode=0
Oct 30 22:10:58.814762 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.814740    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"
Oct 30 22:10:58.814762 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.814741    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" event={"ID":"cfb6383b-3962-43f7-be89-29de959289ea","Type":"ContainerDied","Data":"365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4"}
Oct 30 22:10:58.815080 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.814858    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm" event={"ID":"cfb6383b-3962-43f7-be89-29de959289ea","Type":"ContainerDied","Data":"68203ea36e264ee9963c5735f38775e04f22118cea89e875e8929cc166c8fdc8"}
Oct 30 22:10:58.815080 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.814902    5886 scope.go:122] "RemoveContainer" containerID="365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4"
Oct 30 22:10:58.816668 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.816644    5886 generic.go:356] "Generic (PLEG): container finished" podID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerID="722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8" exitCode=0
Oct 30 22:10:58.816775 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.816714    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" event={"ID":"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e","Type":"ContainerDied","Data":"722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8"}
Oct 30 22:10:58.816775 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.816751    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2" event={"ID":"e5962f18-d7e2-4dc4-83c9-a1f3d784b59e","Type":"ContainerDied","Data":"624da93142e28133bdd20d9f58f97c23f229cecc1d4155def79f855dab10edec"}
Oct 30 22:10:58.816899 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.816817    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"
Oct 30 22:10:58.829133 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.829108    5886 scope.go:122] "RemoveContainer" containerID="365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4"
Oct 30 22:10:58.829491 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:10:58.829471    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4\": container with ID starting with 365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4 not found: ID does not exist" containerID="365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4"
Oct 30 22:10:58.829561 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.829503    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4"} err="failed to get container status \"365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4\": rpc error: code = NotFound desc = could not find container \"365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4\": container with ID starting with 365a5ec946689e06bfb7b87a25e63cd2c3028ea8351fe0534db61ee62d7ec9a4 not found: ID does not exist"
Oct 30 22:10:58.829561 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.829522    5886 scope.go:122] "RemoveContainer" containerID="722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8"
Oct 30 22:10:58.845991 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.842508    5886 scope.go:122] "RemoveContainer" containerID="722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8"
Oct 30 22:10:58.845991 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:10:58.843023    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8\": container with ID starting with 722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8 not found: ID does not exist" containerID="722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8"
Oct 30 22:10:58.845991 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.843086    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8"} err="failed to get container status \"722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8\": rpc error: code = NotFound desc = could not find container \"722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8\": container with ID starting with 722bfcf60856adf57e06248c1889e482d4ecc1f747d28440b4a6b1df955114d8 not found: ID does not exist"
Oct 30 22:10:58.879626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.879596    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"]
Oct 30 22:10:58.903022 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.902990    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9d677d-openshift-default-fd779cd67-rcsl2"]
Oct 30 22:10:58.957529 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.957491    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"]
Oct 30 22:10:58.964769 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:58.964731    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-50ab77-openshift-default-668bc69bbc-rcthm"]
Oct 30 22:10:59.989913 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:59.989883    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb6383b-3962-43f7-be89-29de959289ea" path="/var/lib/kubelet/pods/cfb6383b-3962-43f7-be89-29de959289ea/volumes"
Oct 30 22:10:59.990363 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:10:59.990346    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" path="/var/lib/kubelet/pods/e5962f18-d7e2-4dc4-83c9-a1f3d784b59e/volumes"
Oct 30 22:11:01.835054 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:01.835014    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"]
Oct 30 22:11:01.835505 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:01.835324    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy" containerID="cri-o://2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2" gracePeriod=30
Oct 30 22:11:03.711934 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:03.711886    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"]
Oct 30 22:11:03.712476 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:03.712274    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy" containerID="cri-o://d53d45a09b4e8b72c0b60cd84ed451516a148a579965a20239b806cbc18ffffa" gracePeriod=30
Oct 30 22:11:04.167919 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:04.167877    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"]
Oct 30 22:11:04.168129 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:04.168079    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy" containerID="cri-o://cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b" gracePeriod=30
Oct 30 22:11:06.022587 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:06.022551    5886 patch_prober.go:29] interesting pod/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:06.022982 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:06.022619    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:06.277453 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:06.277371    5886 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:06.277453 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:06.277444    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:07.116783 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.116723    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 22:11:07.156854 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.156822    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-podinfo\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.156860    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-credential-socket\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.156913    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-socket\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.156944    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-envoy\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.156969    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-data\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157246 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.157022    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-kube-api-access-bzbkk\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-kube-api-access-bzbkk\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157246 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.157083    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istiod-ca-cert\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157246 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.157225    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-token\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-token\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157415 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.157248    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-credential-socket" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:07.157415 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.157269    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-socket" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:07.157415 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.157305    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-certs\") pod \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\" (UID: \"b9f1c731-26c5-4ec4-abd8-458909a6ff6e\") "
Oct 30 22:11:07.157884 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.157859    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.158011 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.157887    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.158331 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.158139    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istiod-ca-cert" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:07.158331 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.158298    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-certs" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:07.158331 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.158309    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-data" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:07.159460 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.159437    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-envoy" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:07.159587 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.159514    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-token" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:07.159741 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.159726    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-podinfo" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:07.159799 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.159757    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-kube-api-access-bzbkk" pod "b9f1c731-26c5-4ec4-abd8-458909a6ff6e" (UID: "b9f1c731-26c5-4ec4-abd8-458909a6ff6e"). InnerVolumeSpecName "kube-api-access-bzbkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:07.258749 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.258715    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.258749 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.258741    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.258749 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.258751    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.259000 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.258760    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzbkk\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-kube-api-access-bzbkk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.259000 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.258769    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.259000 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.258778    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.259000 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.258788    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f1c731-26c5-4ec4-abd8-458909a6ff6e-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:07.894240 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.894200    5886 generic.go:356] "Generic (PLEG): container finished" podID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerID="2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2" exitCode=0
Oct 30 22:11:07.894425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.894252    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" event={"ID":"b9f1c731-26c5-4ec4-abd8-458909a6ff6e","Type":"ContainerDied","Data":"2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2"}
Oct 30 22:11:07.894425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.894295    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"
Oct 30 22:11:07.894425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.894352    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj" event={"ID":"b9f1c731-26c5-4ec4-abd8-458909a6ff6e","Type":"ContainerDied","Data":"366727466fe6aa41b437fe020cbced50653b7560bf7a3467d07c8d5c733adfc9"}
Oct 30 22:11:07.894425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.894385    5886 scope.go:122] "RemoveContainer" containerID="2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2"
Oct 30 22:11:07.906603 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.906576    5886 scope.go:122] "RemoveContainer" containerID="2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2"
Oct 30 22:11:07.906908 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:11:07.906884    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2\": container with ID starting with 2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2 not found: ID does not exist" containerID="2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2"
Oct 30 22:11:07.906967 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.906921    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2"} err="failed to get container status \"2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2\": rpc error: code = NotFound desc = could not find container \"2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2\": container with ID starting with 2888ab4ebd6e8927e614bb454142309c1638819075640fa101800779de86f9f2 not found: ID does not exist"
Oct 30 22:11:07.966195 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.966140    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"]
Oct 30 22:11:07.976301 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.976269    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-805edc-openshift-default-795c79ddd5-vx2tj"]
Oct 30 22:11:07.991412 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:07.991383    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" path="/var/lib/kubelet/pods/b9f1c731-26c5-4ec4-abd8-458909a6ff6e/volumes"
Oct 30 22:11:08.909335 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:08.905653    5886 generic.go:356] "Generic (PLEG): container finished" podID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerID="d53d45a09b4e8b72c0b60cd84ed451516a148a579965a20239b806cbc18ffffa" exitCode=0
Oct 30 22:11:08.909335 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:08.905723    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" event={"ID":"57b1ff52-2fef-4268-93b4-52325d4db4ff","Type":"ContainerDied","Data":"d53d45a09b4e8b72c0b60cd84ed451516a148a579965a20239b806cbc18ffffa"}
Oct 30 22:11:09.031471 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.031443    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 22:11:09.082762 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.082717    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-envoy\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.083327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.082804    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-data\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.083327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.082834    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-podinfo\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.083327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.082859    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-token\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-token\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.083327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.082894    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-kube-api-access-qt9xw\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-kube-api-access-qt9xw\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.083327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.082941    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-socket\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.083327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.083042    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-certs\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.083327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.083131    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-credential-socket\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.083327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.083256    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b1ff52-2fef-4268-93b4-52325d4db4ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b1ff52-2fef-4268-93b4-52325d4db4ff-istiod-ca-cert\") pod \"57b1ff52-2fef-4268-93b4-52325d4db4ff\" (UID: \"57b1ff52-2fef-4268-93b4-52325d4db4ff\") "
Oct 30 22:11:09.084038 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.083966    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-credential-socket" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.084146 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.084122    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-data" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.084252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.084194    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-certs" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.084252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.084238    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-socket" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.084379 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.084265    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b1ff52-2fef-4268-93b4-52325d4db4ff-istiod-ca-cert" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:09.085382 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.085295    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.085382 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.085321    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.085382 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.085335    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.085382 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.085349    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.086272 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.086245    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-envoy" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.086734 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.086711    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-kube-api-access-qt9xw" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "kube-api-access-qt9xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:09.086824 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.086787    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-podinfo" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:09.086824 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.086801    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-token" pod "57b1ff52-2fef-4268-93b4-52325d4db4ff" (UID: "57b1ff52-2fef-4268-93b4-52325d4db4ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:09.186261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.186231    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.186378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.186263    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.186378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.186277    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.186378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.186289    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt9xw\" (UniqueName: \"kubernetes.io/projected/57b1ff52-2fef-4268-93b4-52325d4db4ff-kube-api-access-qt9xw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.186378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.186303    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57b1ff52-2fef-4268-93b4-52325d4db4ff-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.420960 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.420931    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:11:09.489852 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.489812    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-token\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-token\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.490031 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.489877    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-envoy\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.490031 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.489911    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-kube-api-access-jpn5t\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-kube-api-access-jpn5t\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.490031 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.489938    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-credential-socket\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.490031 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.489979    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-socket\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.490310 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.490054    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-certs\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.490310 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.490073    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istiod-ca-cert\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.490310 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.490268    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-podinfo\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.490310 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.490300    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-data\") pod \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\" (UID: \"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d\") "
Oct 30 22:11:09.491469 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.491094    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-credential-socket" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.491469 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.491193    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-certs" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.491469 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.491285    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-socket" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.491895 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.491858    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-data" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.491970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.491901    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istiod-ca-cert" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:09.494062 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.494038    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-kube-api-access-jpn5t" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "kube-api-access-jpn5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:09.494213 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.494132    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-podinfo" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:09.494327 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.494292    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-token" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:09.495861 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.495833    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-envoy" pod "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" (UID: "89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592398    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592426    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592437    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592447    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592454    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592462    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592470    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592480    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpn5t\" (UniqueName: \"kubernetes.io/projected/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-kube-api-access-jpn5t\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.592486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.592489    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:09.915742 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.915706    5886 generic.go:356] "Generic (PLEG): container finished" podID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerID="cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b" exitCode=0
Oct 30 22:11:09.916276 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.915767    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"
Oct 30 22:11:09.916276 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.915787    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" event={"ID":"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d","Type":"ContainerDied","Data":"cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b"}
Oct 30 22:11:09.916276 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.915918    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7" event={"ID":"89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d","Type":"ContainerDied","Data":"ec570d253abfa9f11b09d5b3a5847f74c4629f3862b17fd10fc4da72563c088c"}
Oct 30 22:11:09.916276 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.915971    5886 scope.go:122] "RemoveContainer" containerID="cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b"
Oct 30 22:11:09.917510 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.917402    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"
Oct 30 22:11:09.917510 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.917445    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk" event={"ID":"57b1ff52-2fef-4268-93b4-52325d4db4ff","Type":"ContainerDied","Data":"42da3d2a4f866814c54fbe5c08811a72aa068b44df57b16ab4fda6697af65de7"}
Oct 30 22:11:09.927860 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.927841    5886 scope.go:122] "RemoveContainer" containerID="cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b"
Oct 30 22:11:09.928269 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:11:09.928136    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b\": container with ID starting with cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b not found: ID does not exist" containerID="cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b"
Oct 30 22:11:09.928269 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.928201    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b"} err="failed to get container status \"cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b\": rpc error: code = NotFound desc = could not find container \"cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b\": container with ID starting with cbd865a2cef25de9af6d6c2ae8d6d8e0beb30acef910b5a68162ba43d1c1e28b not found: ID does not exist"
Oct 30 22:11:09.928269 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.928224    5886 scope.go:122] "RemoveContainer" containerID="d53d45a09b4e8b72c0b60cd84ed451516a148a579965a20239b806cbc18ffffa"
Oct 30 22:11:09.973626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.973589    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"]
Oct 30 22:11:09.991245 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:09.991207    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ae6492-openshift-default-69b787678c-xvkq7"]
Oct 30 22:11:10.017812 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:10.017778    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"]
Oct 30 22:11:10.022165 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:10.022119    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5494e8-openshift-default-555547d96f-784wk"]
Oct 30 22:11:11.028145 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:11:11.028105    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:11:11.992243 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:11.992207    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" path="/var/lib/kubelet/pods/57b1ff52-2fef-4268-93b4-52325d4db4ff/volumes"
Oct 30 22:11:11.992906 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:11.992882    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" path="/var/lib/kubelet/pods/89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d/volumes"
Oct 30 22:11:34.309519 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.309478    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"]
Oct 30 22:11:34.310264 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.309763    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg" podUID="ff58ecb3-8871-46a9-aa79-a5067e38bb98" containerName="maas-api" containerID="cri-o://26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08" gracePeriod=30
Oct 30 22:11:34.607294 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.607262    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:11:34.731658 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.731624    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff58ecb3-8871-46a9-aa79-a5067e38bb98-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff58ecb3-8871-46a9-aa79-a5067e38bb98-openshift-service-ca\") pod \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") "
Oct 30 22:11:34.731835 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.731728    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-metrics-certs\") pod \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") "
Oct 30 22:11:34.731895 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.731884    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff58ecb3-8871-46a9-aa79-a5067e38bb98-kube-api-access-hc8l2\" (UniqueName: \"kubernetes.io/projected/ff58ecb3-8871-46a9-aa79-a5067e38bb98-kube-api-access-hc8l2\") pod \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") "
Oct 30 22:11:34.731934 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.731910    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-maas-api-tls\") pod \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\" (UID: \"ff58ecb3-8871-46a9-aa79-a5067e38bb98\") "
Oct 30 22:11:34.732071 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.732041    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff58ecb3-8871-46a9-aa79-a5067e38bb98-openshift-service-ca" pod "ff58ecb3-8871-46a9-aa79-a5067e38bb98" (UID: "ff58ecb3-8871-46a9-aa79-a5067e38bb98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:34.732346 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.732319    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff58ecb3-8871-46a9-aa79-a5067e38bb98-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:34.733943 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.733921    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-metrics-certs" pod "ff58ecb3-8871-46a9-aa79-a5067e38bb98" (UID: "ff58ecb3-8871-46a9-aa79-a5067e38bb98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:34.734023 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.733984    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff58ecb3-8871-46a9-aa79-a5067e38bb98-kube-api-access-hc8l2" pod "ff58ecb3-8871-46a9-aa79-a5067e38bb98" (UID: "ff58ecb3-8871-46a9-aa79-a5067e38bb98"). InnerVolumeSpecName "kube-api-access-hc8l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:34.734134 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.734116    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-maas-api-tls" pod "ff58ecb3-8871-46a9-aa79-a5067e38bb98" (UID: "ff58ecb3-8871-46a9-aa79-a5067e38bb98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:34.839773 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.839066    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8l2\" (UniqueName: \"kubernetes.io/projected/ff58ecb3-8871-46a9-aa79-a5067e38bb98-kube-api-access-hc8l2\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:34.839773 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.839105    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:34.839773 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:34.839119    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff58ecb3-8871-46a9-aa79-a5067e38bb98-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:11:35.107985 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.107896    5886 generic.go:356] "Generic (PLEG): container finished" podID="ff58ecb3-8871-46a9-aa79-a5067e38bb98" containerID="26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08" exitCode=0
Oct 30 22:11:35.107985 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.107945    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"
Oct 30 22:11:35.108205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.107981    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg" event={"ID":"ff58ecb3-8871-46a9-aa79-a5067e38bb98","Type":"ContainerDied","Data":"26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08"}
Oct 30 22:11:35.108205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.108046    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg" event={"ID":"ff58ecb3-8871-46a9-aa79-a5067e38bb98","Type":"ContainerDied","Data":"c7df1216aa9ab407caeabf470e9a757a69e48aa6304c3bffd939cbdd1c45ff39"}
Oct 30 22:11:35.108205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.108073    5886 scope.go:122] "RemoveContainer" containerID="26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08"
Oct 30 22:11:35.119240 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.119218    5886 scope.go:122] "RemoveContainer" containerID="26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08"
Oct 30 22:11:35.119504 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:11:35.119482    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08\": container with ID starting with 26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08 not found: ID does not exist" containerID="26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08"
Oct 30 22:11:35.119573 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.119512    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08"} err="failed to get container status \"26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08\": rpc error: code = NotFound desc = could not find container \"26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08\": container with ID starting with 26870ebae4ed62711b4979695a1787c1f5f753cca604c576ed9a7aba80c06d08 not found: ID does not exist"
Oct 30 22:11:35.158350 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.158326    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"]
Oct 30 22:11:35.162815 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.162789    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d53e0-54796b7cf-ghfwg"]
Oct 30 22:11:35.988973 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:11:35.988934    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff58ecb3-8871-46a9-aa79-a5067e38bb98" path="/var/lib/kubelet/pods/ff58ecb3-8871-46a9-aa79-a5067e38bb98/volumes"
Oct 30 22:11:41.029330 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:11:41.029246    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:12:11.029766 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:12:11.029727    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:12:15.533879 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.533842    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"]
Oct 30 22:12:15.534666 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.534101    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr" podUID="7ed45761-7d92-4890-9405-3743f30a43bb" containerName="maas-api" containerID="cri-o://5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333" gracePeriod=30
Oct 30 22:12:15.853415 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.853377    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:12:15.900076 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.899213    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed45761-7d92-4890-9405-3743f30a43bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed45761-7d92-4890-9405-3743f30a43bb-openshift-service-ca\") pod \"7ed45761-7d92-4890-9405-3743f30a43bb\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") "
Oct 30 22:12:15.900076 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.899254    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed45761-7d92-4890-9405-3743f30a43bb-kube-api-access-hz6dw\" (UniqueName: \"kubernetes.io/projected/7ed45761-7d92-4890-9405-3743f30a43bb-kube-api-access-hz6dw\") pod \"7ed45761-7d92-4890-9405-3743f30a43bb\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") "
Oct 30 22:12:15.900076 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.899304    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-maas-api-tls\") pod \"7ed45761-7d92-4890-9405-3743f30a43bb\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") "
Oct 30 22:12:15.900076 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.899390    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-metrics-certs\") pod \"7ed45761-7d92-4890-9405-3743f30a43bb\" (UID: \"7ed45761-7d92-4890-9405-3743f30a43bb\") "
Oct 30 22:12:15.901062 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.901033    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed45761-7d92-4890-9405-3743f30a43bb-openshift-service-ca" pod "7ed45761-7d92-4890-9405-3743f30a43bb" (UID: "7ed45761-7d92-4890-9405-3743f30a43bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:15.904030 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.903991    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed45761-7d92-4890-9405-3743f30a43bb-kube-api-access-hz6dw" pod "7ed45761-7d92-4890-9405-3743f30a43bb" (UID: "7ed45761-7d92-4890-9405-3743f30a43bb"). InnerVolumeSpecName "kube-api-access-hz6dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:15.904575 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.904092    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-maas-api-tls" pod "7ed45761-7d92-4890-9405-3743f30a43bb" (UID: "7ed45761-7d92-4890-9405-3743f30a43bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:15.904575 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:15.904475    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-metrics-certs" pod "7ed45761-7d92-4890-9405-3743f30a43bb" (UID: "7ed45761-7d92-4890-9405-3743f30a43bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:16.000550 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.000513    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed45761-7d92-4890-9405-3743f30a43bb-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:16.000550 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.000543    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz6dw\" (UniqueName: \"kubernetes.io/projected/7ed45761-7d92-4890-9405-3743f30a43bb-kube-api-access-hz6dw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:16.000550 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.000554    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:16.000550 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.000562    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed45761-7d92-4890-9405-3743f30a43bb-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:16.426825 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.426789    5886 generic.go:356] "Generic (PLEG): container finished" podID="7ed45761-7d92-4890-9405-3743f30a43bb" containerID="5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333" exitCode=0
Oct 30 22:12:16.427025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.426835    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"
Oct 30 22:12:16.427025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.426877    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr" event={"ID":"7ed45761-7d92-4890-9405-3743f30a43bb","Type":"ContainerDied","Data":"5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333"}
Oct 30 22:12:16.427025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.426940    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr" event={"ID":"7ed45761-7d92-4890-9405-3743f30a43bb","Type":"ContainerDied","Data":"788075af98ecea530ceb40e44e8c08ff3521184cfd384ce72074368cfc6f5c52"}
Oct 30 22:12:16.427025 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.426954    5886 scope.go:122] "RemoveContainer" containerID="5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333"
Oct 30 22:12:16.437990 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.437962    5886 scope.go:122] "RemoveContainer" containerID="5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333"
Oct 30 22:12:16.438282 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:12:16.438256    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333\": container with ID starting with 5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333 not found: ID does not exist" containerID="5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333"
Oct 30 22:12:16.438373 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.438292    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333"} err="failed to get container status \"5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333\": rpc error: code = NotFound desc = could not find container \"5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333\": container with ID starting with 5e4b4326d7a00bc9fc5c3cc44cc81395e6d283c0de11aadd338d91098e3d1333 not found: ID does not exist"
Oct 30 22:12:16.479241 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.479210    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"]
Oct 30 22:12:16.490388 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:16.490359    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6537e6-58bb7f5fb5-r7qcr"]
Oct 30 22:12:17.991622 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:17.991583    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed45761-7d92-4890-9405-3743f30a43bb" path="/var/lib/kubelet/pods/7ed45761-7d92-4890-9405-3743f30a43bb/volumes"
Oct 30 22:12:41.028294 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:12:41.028258    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:12:43.481476 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:43.481432    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"]
Oct 30 22:12:47.849041 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:47.849006    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"]
Oct 30 22:12:47.849559 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:47.849221    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m" podUID="745272f1-fc0f-45ee-9e94-c5c9fc2583ed" containerName="maas-api" containerID="cri-o://526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119" gracePeriod=30
Oct 30 22:12:48.373866 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.373837    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:12:48.482621 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.482511    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk" podUID="5a39adb4-8217-40e0-9e5a-843e148a49c5" containerName="payload-pre-processing" containerID="cri-o://9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90" gracePeriod=25
Oct 30 22:12:48.535228 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.535197    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-metrics-certs\") pod \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") "
Oct 30 22:12:48.535399 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.535276    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-maas-api-tls\") pod \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") "
Oct 30 22:12:48.535399 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.535318    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-kube-api-access-4w4pd\" (UniqueName: \"kubernetes.io/projected/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-kube-api-access-4w4pd\") pod \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") "
Oct 30 22:12:48.535399 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.535388    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-openshift-service-ca\") pod \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\" (UID: \"745272f1-fc0f-45ee-9e94-c5c9fc2583ed\") "
Oct 30 22:12:48.535908 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.535862    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-openshift-service-ca" pod "745272f1-fc0f-45ee-9e94-c5c9fc2583ed" (UID: "745272f1-fc0f-45ee-9e94-c5c9fc2583ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:48.537560 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.537513    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-metrics-certs" pod "745272f1-fc0f-45ee-9e94-c5c9fc2583ed" (UID: "745272f1-fc0f-45ee-9e94-c5c9fc2583ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:48.537698 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.537589    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-maas-api-tls" pod "745272f1-fc0f-45ee-9e94-c5c9fc2583ed" (UID: "745272f1-fc0f-45ee-9e94-c5c9fc2583ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:48.537698 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.537607    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-kube-api-access-4w4pd" pod "745272f1-fc0f-45ee-9e94-c5c9fc2583ed" (UID: "745272f1-fc0f-45ee-9e94-c5c9fc2583ed"). InnerVolumeSpecName "kube-api-access-4w4pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:48.637617 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.637583    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.637617 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.637616    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.637839 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.637661    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4pd\" (UniqueName: \"kubernetes.io/projected/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-kube-api-access-4w4pd\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.637839 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.637676    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/745272f1-fc0f-45ee-9e94-c5c9fc2583ed-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:48.724482 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.724441    5886 generic.go:356] "Generic (PLEG): container finished" podID="745272f1-fc0f-45ee-9e94-c5c9fc2583ed" containerID="526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119" exitCode=0
Oct 30 22:12:48.724641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.724491    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"
Oct 30 22:12:48.724641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.724529    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m" event={"ID":"745272f1-fc0f-45ee-9e94-c5c9fc2583ed","Type":"ContainerDied","Data":"526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119"}
Oct 30 22:12:48.724641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.724588    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m" event={"ID":"745272f1-fc0f-45ee-9e94-c5c9fc2583ed","Type":"ContainerDied","Data":"ede7f30ef47408c319dff06f4a5fdefa144802e0934bf1205c8e40a27a3642b4"}
Oct 30 22:12:48.724641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.724611    5886 scope.go:122] "RemoveContainer" containerID="526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119"
Oct 30 22:12:48.735356 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.735339    5886 scope.go:122] "RemoveContainer" containerID="526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119"
Oct 30 22:12:48.735612 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:12:48.735591    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119\": container with ID starting with 526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119 not found: ID does not exist" containerID="526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119"
Oct 30 22:12:48.735675 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.735620    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119"} err="failed to get container status \"526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119\": rpc error: code = NotFound desc = could not find container \"526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119\": container with ID starting with 526169ecbfa2edf84790a3d8757f3f1bce11d4e8dbda895ddff228333a15f119 not found: ID does not exist"
Oct 30 22:12:48.790179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.789730    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"]
Oct 30 22:12:48.805808 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:48.805768    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a42c31-55f58cd9d8-smq9m"]
Oct 30 22:12:49.988634 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:49.988591    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745272f1-fc0f-45ee-9e94-c5c9fc2583ed" path="/var/lib/kubelet/pods/745272f1-fc0f-45ee-9e94-c5c9fc2583ed/volumes"
Oct 30 22:12:50.201705 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.201675    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"]
Oct 30 22:12:50.201927 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.201899    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7" podUID="1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" containerName="maas-api" containerID="cri-o://6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb" gracePeriod=30
Oct 30 22:12:50.536985 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.536955    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:12:50.577172 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.577065    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls\") pod \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") "
Oct 30 22:12:50.583170 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.583109    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-kube-api-access-fzpmh\" (UniqueName: \"kubernetes.io/projected/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-kube-api-access-fzpmh\") pod \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") "
Oct 30 22:12:50.591628 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.591590    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-openshift-service-ca\") pod \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") "
Oct 30 22:12:50.591628 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.591668    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-metrics-certs\") pod \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\" (UID: \"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e\") "
Oct 30 22:12:50.592041 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.592009    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-openshift-service-ca" pod "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" (UID: "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:50.596388 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.596359    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:50.643179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.598877    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-metrics-certs" pod "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" (UID: "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:50.643179 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.606314    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls" pod "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" (UID: "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:50.644757 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.644692    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-kube-api-access-fzpmh" pod "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" (UID: "1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e"). InnerVolumeSpecName "kube-api-access-fzpmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:50.728122 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.728091    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzpmh\" (UniqueName: \"kubernetes.io/projected/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-kube-api-access-fzpmh\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:50.728122 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.728119    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:50.728122 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.728129    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:12:50.741270 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.741227    5886 generic.go:356] "Generic (PLEG): container finished" podID="1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" containerID="6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb" exitCode=0
Oct 30 22:12:50.741422 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.741272    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7" event={"ID":"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e","Type":"ContainerDied","Data":"6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb"}
Oct 30 22:12:50.741422 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.741304    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"
Oct 30 22:12:50.741422 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.741347    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7" event={"ID":"1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e","Type":"ContainerDied","Data":"529e382d2ef88f9f9def81dd3914b398316a8f9457d5052d7ef1ce273d0a20ad"}
Oct 30 22:12:50.741422 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.741379    5886 scope.go:122] "RemoveContainer" containerID="6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb"
Oct 30 22:12:50.753220 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.753199    5886 scope.go:122] "RemoveContainer" containerID="6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb"
Oct 30 22:12:50.753509 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:12:50.753488    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb\": container with ID starting with 6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb not found: ID does not exist" containerID="6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb"
Oct 30 22:12:50.753596 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.753518    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb"} err="failed to get container status \"6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb\": rpc error: code = NotFound desc = could not find container \"6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb\": container with ID starting with 6a7fef7cbce9733421c45f256b332b8b172219b09944aedca32ca16ec2fb46cb not found: ID does not exist"
Oct 30 22:12:50.803702 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.803650    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"]
Oct 30 22:12:50.807065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:50.807029    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5454af-977786845-4hcf7"]
Oct 30 22:12:51.155095 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:51.155056    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk" podUID="5a39adb4-8217-40e0-9e5a-843e148a49c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:51.989287 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:12:51.989255    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" path="/var/lib/kubelet/pods/1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e/volumes"
Oct 30 22:13:01.154930 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:01.154893    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk" podUID="5a39adb4-8217-40e0-9e5a-843e148a49c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:04.863956 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:04.863914    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"]
Oct 30 22:13:04.896325 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:04.896286    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"]
Oct 30 22:13:06.976396 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:06.976363    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"]
Oct 30 22:13:08.748195 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.748098    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:13:08.887381 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.887347    5886 generic.go:356] "Generic (PLEG): container finished" podID="5a39adb4-8217-40e0-9e5a-843e148a49c5" containerID="9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90" exitCode=0
Oct 30 22:13:08.887535 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.887396    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"
Oct 30 22:13:08.887535 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.887431    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk" event={"ID":"5a39adb4-8217-40e0-9e5a-843e148a49c5","Type":"ContainerDied","Data":"9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90"}
Oct 30 22:13:08.887535 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.887481    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk" event={"ID":"5a39adb4-8217-40e0-9e5a-843e148a49c5","Type":"ContainerDied","Data":"bc7c54dba4f2f609771c8732a3f7fa3b23ff03098324b4ebf3267f14078fdcf4"}
Oct 30 22:13:08.887535 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.887504    5886 scope.go:122] "RemoveContainer" containerID="9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90"
Oct 30 22:13:08.888616 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.888488    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a39adb4-8217-40e0-9e5a-843e148a49c5-kube-api-access-9v9j5\" (UniqueName: \"kubernetes.io/projected/5a39adb4-8217-40e0-9e5a-843e148a49c5-kube-api-access-9v9j5\") pod \"5a39adb4-8217-40e0-9e5a-843e148a49c5\" (UID: \"5a39adb4-8217-40e0-9e5a-843e148a49c5\") "
Oct 30 22:13:08.888753 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.888738    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a39adb4-8217-40e0-9e5a-843e148a49c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a39adb4-8217-40e0-9e5a-843e148a49c5-plugins-config-volume\") pod \"5a39adb4-8217-40e0-9e5a-843e148a49c5\" (UID: \"5a39adb4-8217-40e0-9e5a-843e148a49c5\") "
Oct 30 22:13:08.889197 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.889139    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a39adb4-8217-40e0-9e5a-843e148a49c5-plugins-config-volume" pod "5a39adb4-8217-40e0-9e5a-843e148a49c5" (UID: "5a39adb4-8217-40e0-9e5a-843e148a49c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:08.890659 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.890637    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a39adb4-8217-40e0-9e5a-843e148a49c5-kube-api-access-9v9j5" pod "5a39adb4-8217-40e0-9e5a-843e148a49c5" (UID: "5a39adb4-8217-40e0-9e5a-843e148a49c5"). InnerVolumeSpecName "kube-api-access-9v9j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:08.906018 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.905997    5886 scope.go:122] "RemoveContainer" containerID="9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90"
Oct 30 22:13:08.906345 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:08.906327    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90\": container with ID starting with 9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90 not found: ID does not exist" containerID="9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90"
Oct 30 22:13:08.906393 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.906353    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90"} err="failed to get container status \"9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90\": rpc error: code = NotFound desc = could not find container \"9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90\": container with ID starting with 9741974e57134f1521972222db10df02abda79b496260f62706c1cf212d02e90 not found: ID does not exist"
Oct 30 22:13:08.990173 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.990130    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a39adb4-8217-40e0-9e5a-843e148a49c5-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:08.990173 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:08.990176    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v9j5\" (UniqueName: \"kubernetes.io/projected/5a39adb4-8217-40e0-9e5a-843e148a49c5-kube-api-access-9v9j5\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:09.232600 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:09.232563    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"]
Oct 30 22:13:09.237771 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:09.237741    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a42c31-66b7f8c779-zzgtk"]
Oct 30 22:13:09.864368 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:09.864314    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd" podUID="53d2c15f-71ee-4335-bf9d-50632aab2c62" containerName="payload-processing" containerID="cri-o://af87f71271b08ba18228580e6ea68b928d811e8cf6eb2d36d14ed91d212c3105" gracePeriod=25
Oct 30 22:13:09.896875 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:09.896830    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md" podUID="98aee083-5553-48d7-a3d9-451c5588e455" containerName="payload-pre-processing" containerID="cri-o://4729909e56ca9227467acab15e176c75a346dc2dae1832e2e9ff21b298a91a9b" gracePeriod=25
Oct 30 22:13:09.988925 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:09.988879    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a39adb4-8217-40e0-9e5a-843e148a49c5" path="/var/lib/kubelet/pods/5a39adb4-8217-40e0-9e5a-843e148a49c5/volumes"
Oct 30 22:13:11.028576 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:11.028494    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:13:11.976792 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:11.976745    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9" podUID="21651e7e-fce1-4e60-b50a-20b0ce4f77b9" containerName="payload-processing" containerID="cri-o://9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e" gracePeriod=25
Oct 30 22:13:14.431302 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:14.431260    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9" podUID="21651e7e-fce1-4e60-b50a-20b0ce4f77b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:15.959143 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:15.959102    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md" podUID="98aee083-5553-48d7-a3d9-451c5588e455" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:15.959649 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:15.959103    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd" podUID="53d2c15f-71ee-4335-bf9d-50632aab2c62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:21.177846 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.177797    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"]
Oct 30 22:13:21.178859 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.178778    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q" podUID="4ccd3396-e2ef-4338-aca4-3d3fe3296910" containerName="maas-api" containerID="cri-o://c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc" gracePeriod=30
Oct 30 22:13:21.459988 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.459961    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:13:21.481534 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.481500    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-maas-api-tls\") pod \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") "
Oct 30 22:13:21.481693 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.481546    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccd3396-e2ef-4338-aca4-3d3fe3296910-kube-api-access-xsdvf\" (UniqueName: \"kubernetes.io/projected/4ccd3396-e2ef-4338-aca4-3d3fe3296910-kube-api-access-xsdvf\") pod \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") "
Oct 30 22:13:21.481693 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.481657    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccd3396-e2ef-4338-aca4-3d3fe3296910-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccd3396-e2ef-4338-aca4-3d3fe3296910-openshift-service-ca\") pod \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") "
Oct 30 22:13:21.481813 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.481724    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-metrics-certs\") pod \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\" (UID: \"4ccd3396-e2ef-4338-aca4-3d3fe3296910\") "
Oct 30 22:13:21.482076 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.482044    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccd3396-e2ef-4338-aca4-3d3fe3296910-openshift-service-ca" pod "4ccd3396-e2ef-4338-aca4-3d3fe3296910" (UID: "4ccd3396-e2ef-4338-aca4-3d3fe3296910"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:21.482278 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.482253    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccd3396-e2ef-4338-aca4-3d3fe3296910-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:21.483776 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.483749    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccd3396-e2ef-4338-aca4-3d3fe3296910-kube-api-access-xsdvf" pod "4ccd3396-e2ef-4338-aca4-3d3fe3296910" (UID: "4ccd3396-e2ef-4338-aca4-3d3fe3296910"). InnerVolumeSpecName "kube-api-access-xsdvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:21.483776 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.483769    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-maas-api-tls" pod "4ccd3396-e2ef-4338-aca4-3d3fe3296910" (UID: "4ccd3396-e2ef-4338-aca4-3d3fe3296910"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:21.483910 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.483751    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-metrics-certs" pod "4ccd3396-e2ef-4338-aca4-3d3fe3296910" (UID: "4ccd3396-e2ef-4338-aca4-3d3fe3296910"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:21.585202 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.583539    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:21.585202 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.583578    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccd3396-e2ef-4338-aca4-3d3fe3296910-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:21.585202 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.583593    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdvf\" (UniqueName: \"kubernetes.io/projected/4ccd3396-e2ef-4338-aca4-3d3fe3296910-kube-api-access-xsdvf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:21.979876 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.979840    5886 generic.go:356] "Generic (PLEG): container finished" podID="4ccd3396-e2ef-4338-aca4-3d3fe3296910" containerID="c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc" exitCode=0
Oct 30 22:13:21.980089 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.979896    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"
Oct 30 22:13:21.980089 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.979913    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q" event={"ID":"4ccd3396-e2ef-4338-aca4-3d3fe3296910","Type":"ContainerDied","Data":"c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc"}
Oct 30 22:13:21.980089 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.979964    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q" event={"ID":"4ccd3396-e2ef-4338-aca4-3d3fe3296910","Type":"ContainerDied","Data":"a3aa8f2a0c8583396b76c8d27eedfb1dbfcafb0fa5009a9d1986d10ec5f8fea6"}
Oct 30 22:13:21.980089 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.979990    5886 scope.go:122] "RemoveContainer" containerID="c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc"
Oct 30 22:13:21.993684 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.993660    5886 scope.go:122] "RemoveContainer" containerID="c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc"
Oct 30 22:13:21.993971 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:21.993954    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc\": container with ID starting with c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc not found: ID does not exist" containerID="c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc"
Oct 30 22:13:21.994032 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:21.993977    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc"} err="failed to get container status \"c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc\": rpc error: code = NotFound desc = could not find container \"c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc\": container with ID starting with c4a692cb7a01c6b12f1f95ade00b3f55d02005efc5460c798827c5e73679cffc not found: ID does not exist"
Oct 30 22:13:22.087190 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:22.085550    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"]
Oct 30 22:13:22.091820 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:22.091782    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-589ebc-6769f4877c-fcj5q"]
Oct 30 22:13:23.988241 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:23.988207    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccd3396-e2ef-4338-aca4-3d3fe3296910" path="/var/lib/kubelet/pods/4ccd3396-e2ef-4338-aca4-3d3fe3296910/volumes"
Oct 30 22:13:24.431005 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:24.430966    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9" podUID="21651e7e-fce1-4e60-b50a-20b0ce4f77b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:25.959048 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:25.959006    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd" podUID="53d2c15f-71ee-4335-bf9d-50632aab2c62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:25.959559 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:25.959020    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md" podUID="98aee083-5553-48d7-a3d9-451c5588e455" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:30.036911 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.036865    5886 generic.go:356] "Generic (PLEG): container finished" podID="98aee083-5553-48d7-a3d9-451c5588e455" containerID="4729909e56ca9227467acab15e176c75a346dc2dae1832e2e9ff21b298a91a9b" exitCode=0
Oct 30 22:13:30.037611 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.036969    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md" event={"ID":"98aee083-5553-48d7-a3d9-451c5588e455","Type":"ContainerDied","Data":"4729909e56ca9227467acab15e176c75a346dc2dae1832e2e9ff21b298a91a9b"}
Oct 30 22:13:30.038693 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.038661    5886 generic.go:356] "Generic (PLEG): container finished" podID="53d2c15f-71ee-4335-bf9d-50632aab2c62" containerID="af87f71271b08ba18228580e6ea68b928d811e8cf6eb2d36d14ed91d212c3105" exitCode=0
Oct 30 22:13:30.038866 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.038706    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd" event={"ID":"53d2c15f-71ee-4335-bf9d-50632aab2c62","Type":"ContainerDied","Data":"af87f71271b08ba18228580e6ea68b928d811e8cf6eb2d36d14ed91d212c3105"}
Oct 30 22:13:30.170470 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.170434    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 22:13:30.218826 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.218800    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 22:13:30.309935 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.309898    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98aee083-5553-48d7-a3d9-451c5588e455-kube-api-access-gqrgf\" (UniqueName: \"kubernetes.io/projected/98aee083-5553-48d7-a3d9-451c5588e455-kube-api-access-gqrgf\") pod \"98aee083-5553-48d7-a3d9-451c5588e455\" (UID: \"98aee083-5553-48d7-a3d9-451c5588e455\") "
Oct 30 22:13:30.310125 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.310002    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53d2c15f-71ee-4335-bf9d-50632aab2c62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53d2c15f-71ee-4335-bf9d-50632aab2c62-plugins-config-volume\") pod \"53d2c15f-71ee-4335-bf9d-50632aab2c62\" (UID: \"53d2c15f-71ee-4335-bf9d-50632aab2c62\") "
Oct 30 22:13:30.310125 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.310087    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98aee083-5553-48d7-a3d9-451c5588e455-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98aee083-5553-48d7-a3d9-451c5588e455-plugins-config-volume\") pod \"98aee083-5553-48d7-a3d9-451c5588e455\" (UID: \"98aee083-5553-48d7-a3d9-451c5588e455\") "
Oct 30 22:13:30.310399 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.310125    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53d2c15f-71ee-4335-bf9d-50632aab2c62-kube-api-access-t2fzg\" (UniqueName: \"kubernetes.io/projected/53d2c15f-71ee-4335-bf9d-50632aab2c62-kube-api-access-t2fzg\") pod \"53d2c15f-71ee-4335-bf9d-50632aab2c62\" (UID: \"53d2c15f-71ee-4335-bf9d-50632aab2c62\") "
Oct 30 22:13:30.310664 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.310639    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98aee083-5553-48d7-a3d9-451c5588e455-plugins-config-volume" pod "98aee083-5553-48d7-a3d9-451c5588e455" (UID: "98aee083-5553-48d7-a3d9-451c5588e455"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:30.310781 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.310639    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53d2c15f-71ee-4335-bf9d-50632aab2c62-plugins-config-volume" pod "53d2c15f-71ee-4335-bf9d-50632aab2c62" (UID: "53d2c15f-71ee-4335-bf9d-50632aab2c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:30.312404 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.312374    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53d2c15f-71ee-4335-bf9d-50632aab2c62-kube-api-access-t2fzg" pod "53d2c15f-71ee-4335-bf9d-50632aab2c62" (UID: "53d2c15f-71ee-4335-bf9d-50632aab2c62"). InnerVolumeSpecName "kube-api-access-t2fzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:30.312832 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.312813    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98aee083-5553-48d7-a3d9-451c5588e455-kube-api-access-gqrgf" pod "98aee083-5553-48d7-a3d9-451c5588e455" (UID: "98aee083-5553-48d7-a3d9-451c5588e455"). InnerVolumeSpecName "kube-api-access-gqrgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:30.412359 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.412318    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqrgf\" (UniqueName: \"kubernetes.io/projected/98aee083-5553-48d7-a3d9-451c5588e455-kube-api-access-gqrgf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:30.412359 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.412359    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53d2c15f-71ee-4335-bf9d-50632aab2c62-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:30.412588 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.412379    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98aee083-5553-48d7-a3d9-451c5588e455-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:30.412588 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:30.412394    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2fzg\" (UniqueName: \"kubernetes.io/projected/53d2c15f-71ee-4335-bf9d-50632aab2c62-kube-api-access-t2fzg\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:31.047684 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.047638    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md" event={"ID":"98aee083-5553-48d7-a3d9-451c5588e455","Type":"ContainerDied","Data":"3b94006b24b0fa818b304d1ea41db74b90a99b806e6f9b5672748ace7f42a418"}
Oct 30 22:13:31.048198 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.047716    5886 scope.go:122] "RemoveContainer" containerID="4729909e56ca9227467acab15e176c75a346dc2dae1832e2e9ff21b298a91a9b"
Oct 30 22:13:31.048198 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.047855    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"
Oct 30 22:13:31.050305 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.050264    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd" event={"ID":"53d2c15f-71ee-4335-bf9d-50632aab2c62","Type":"ContainerDied","Data":"eb922a4ef731a25ce2490f3a34c1a93e0d5870f8bf95cdeb3a96c1872c0eb1b0"}
Oct 30 22:13:31.050428 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.050395    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"
Oct 30 22:13:31.060355 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.060228    5886 scope.go:122] "RemoveContainer" containerID="af87f71271b08ba18228580e6ea68b928d811e8cf6eb2d36d14ed91d212c3105"
Oct 30 22:13:31.108796 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.108765    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"]
Oct 30 22:13:31.124270 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.124235    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-846656-7d98bfbbd7-724md"]
Oct 30 22:13:31.226180 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.220772    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"]
Oct 30 22:13:31.226180 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.220840    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-846656-5f8cc96dc-dppkd"]
Oct 30 22:13:31.993875 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.993842    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53d2c15f-71ee-4335-bf9d-50632aab2c62" path="/var/lib/kubelet/pods/53d2c15f-71ee-4335-bf9d-50632aab2c62/volumes"
Oct 30 22:13:31.994470 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:31.994451    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98aee083-5553-48d7-a3d9-451c5588e455" path="/var/lib/kubelet/pods/98aee083-5553-48d7-a3d9-451c5588e455/volumes"
Oct 30 22:13:32.274416 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.274386    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:13:32.367007 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.351464    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-plugins-config-volume\") pod \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\" (UID: \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\") "
Oct 30 22:13:32.367007 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.351571    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-kube-api-access-52656\" (UniqueName: \"kubernetes.io/projected/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-kube-api-access-52656\") pod \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\" (UID: \"21651e7e-fce1-4e60-b50a-20b0ce4f77b9\") "
Oct 30 22:13:32.367007 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.352571    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-plugins-config-volume" pod "21651e7e-fce1-4e60-b50a-20b0ce4f77b9" (UID: "21651e7e-fce1-4e60-b50a-20b0ce4f77b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:32.367007 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.354685    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-kube-api-access-52656" pod "21651e7e-fce1-4e60-b50a-20b0ce4f77b9" (UID: "21651e7e-fce1-4e60-b50a-20b0ce4f77b9"). InnerVolumeSpecName "kube-api-access-52656". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:32.401259 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.401207    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"]
Oct 30 22:13:32.401507 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.401481    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy" containerID="cri-o://420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570" gracePeriod=30
Oct 30 22:13:32.452801 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.452771    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:32.452801 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:32.452798    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52656\" (UniqueName: \"kubernetes.io/projected/21651e7e-fce1-4e60-b50a-20b0ce4f77b9-kube-api-access-52656\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:33.068269 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.068235    5886 generic.go:356] "Generic (PLEG): container finished" podID="21651e7e-fce1-4e60-b50a-20b0ce4f77b9" containerID="9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e" exitCode=0
Oct 30 22:13:33.068440 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.068281    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"
Oct 30 22:13:33.068440 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.068297    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9" event={"ID":"21651e7e-fce1-4e60-b50a-20b0ce4f77b9","Type":"ContainerDied","Data":"9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e"}
Oct 30 22:13:33.068440 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.068339    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9" event={"ID":"21651e7e-fce1-4e60-b50a-20b0ce4f77b9","Type":"ContainerDied","Data":"ae3b6fa1e101d1d195612616b8f49817a32d5085732498aaef1232979e24ecc1"}
Oct 30 22:13:33.068440 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.068361    5886 scope.go:122] "RemoveContainer" containerID="9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e"
Oct 30 22:13:33.080995 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.080969    5886 scope.go:122] "RemoveContainer" containerID="9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e"
Oct 30 22:13:33.081356 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:33.081323    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e\": container with ID starting with 9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e not found: ID does not exist" containerID="9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e"
Oct 30 22:13:33.081425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.081366    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e"} err="failed to get container status \"9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e\": rpc error: code = NotFound desc = could not find container \"9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e\": container with ID starting with 9c8de12267272d740d4e4492af93d21740582903f9da0753cfc37267e6e0489e not found: ID does not exist"
Oct 30 22:13:33.119964 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.119933    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"]
Oct 30 22:13:33.131369 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.131337    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19a725-d9c589989-bj8l9"]
Oct 30 22:13:33.989628 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:33.989597    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21651e7e-fce1-4e60-b50a-20b0ce4f77b9" path="/var/lib/kubelet/pods/21651e7e-fce1-4e60-b50a-20b0ce4f77b9/volumes"
Oct 30 22:13:37.683704 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.683636    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:13:37.818263 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818228    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-kube-api-access-jbqvs\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-kube-api-access-jbqvs\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.818468 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818313    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d282f4-907f-4151-af9b-d5243ddc00ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d282f4-907f-4151-af9b-d5243ddc00ea-istiod-ca-cert\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.818468 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818403    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-podinfo\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.818613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818527    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-credential-socket\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.818613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818567    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-data\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.818733 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818616    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-certs\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.818733 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818648    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-socket\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.818852 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818791    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-token\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-token\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.818852 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818793    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-credential-socket" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:37.818852 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818831    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-data" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:37.819016 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818858    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-certs" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:37.819016 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.818892    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-envoy\") pod \"d4d282f4-907f-4151-af9b-d5243ddc00ea\" (UID: \"d4d282f4-907f-4151-af9b-d5243ddc00ea\") "
Oct 30 22:13:37.819131 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.819012    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-socket" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:37.819379 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.819351    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d282f4-907f-4151-af9b-d5243ddc00ea-istiod-ca-cert" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:37.819626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.819539    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d282f4-907f-4151-af9b-d5243ddc00ea-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.819626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.819563    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.819626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.819579    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.819626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.819593    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.819626 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.819605    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.820746 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.820715    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-podinfo" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:37.821017 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.820994    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-kube-api-access-jbqvs" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "kube-api-access-jbqvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:37.821110 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.821028    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-token" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:37.821110 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.821097    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-envoy" pod "d4d282f4-907f-4151-af9b-d5243ddc00ea" (UID: "d4d282f4-907f-4151-af9b-d5243ddc00ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:37.920738 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.920702    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbqvs\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-kube-api-access-jbqvs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.920738 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.920735    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.920738 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.920745    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:37.920995 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:37.920754    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d282f4-907f-4151-af9b-d5243ddc00ea-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:38.104246 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.104213    5886 generic.go:356] "Generic (PLEG): container finished" podID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerID="420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570" exitCode=0
Oct 30 22:13:38.104455 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.104273    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"
Oct 30 22:13:38.104455 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.104295    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" event={"ID":"d4d282f4-907f-4151-af9b-d5243ddc00ea","Type":"ContainerDied","Data":"420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570"}
Oct 30 22:13:38.104455 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.104350    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj" event={"ID":"d4d282f4-907f-4151-af9b-d5243ddc00ea","Type":"ContainerDied","Data":"7f35d0e4dca96b536b69ac54429a96ec361889455033f004813c3b9966902190"}
Oct 30 22:13:38.104455 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.104375    5886 scope.go:122] "RemoveContainer" containerID="420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570"
Oct 30 22:13:38.114634 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.114602    5886 scope.go:122] "RemoveContainer" containerID="820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628"
Oct 30 22:13:38.124128 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.124110    5886 scope.go:122] "RemoveContainer" containerID="420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570"
Oct 30 22:13:38.124460 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:38.124437    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570\": container with ID starting with 420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570 not found: ID does not exist" containerID="420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570"
Oct 30 22:13:38.124515 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.124466    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570"} err="failed to get container status \"420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570\": rpc error: code = NotFound desc = could not find container \"420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570\": container with ID starting with 420816264dc19ed5f7f4df9734f157bb166e8521617d6d35cc86761043685570 not found: ID does not exist"
Oct 30 22:13:38.124515 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.124484    5886 scope.go:122] "RemoveContainer" containerID="820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628"
Oct 30 22:13:38.124708 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:38.124690    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628\": container with ID starting with 820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628 not found: ID does not exist" containerID="820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628"
Oct 30 22:13:38.124758 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.124713    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628"} err="failed to get container status \"820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628\": rpc error: code = NotFound desc = could not find container \"820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628\": container with ID starting with 820dd9408956d966e8161673e900bf754f2fcb3ff2ec8cb117d64b432ac11628 not found: ID does not exist"
Oct 30 22:13:38.157049 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.157021    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"]
Oct 30 22:13:38.169265 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:38.169235    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-19a725-openshift-default-66b4f446f9-g2jpj"]
Oct 30 22:13:39.988128 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:39.988093    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" path="/var/lib/kubelet/pods/d4d282f4-907f-4151-af9b-d5243ddc00ea/volumes"
Oct 30 22:13:41.001711 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.001668    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"]
Oct 30 22:13:41.002234 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.001910    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q" podUID="415f6ded-e8b1-4004-8c65-fa9aecaf195b" containerName="discovery" containerID="cri-o://08a44ff0d959ff1d27b29c52abf289a0d3c28054ebe6b2caa6df00d3455c1bf7" gracePeriod=30
Oct 30 22:13:41.031850 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:41.031811    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:13:41.080111 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.080079    5886 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6cd5q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:8080/ready\": dial tcp 10.133.0.88:8080: connect: connection refused" start-of-body=
Oct 30 22:13:41.080267 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.080149    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q" podUID="415f6ded-e8b1-4004-8c65-fa9aecaf195b" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.88:8080/ready\": dial tcp 10.133.0.88:8080: connect: connection refused"
Oct 30 22:13:41.128986 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.128956    5886 generic.go:356] "Generic (PLEG): container finished" podID="415f6ded-e8b1-4004-8c65-fa9aecaf195b" containerID="08a44ff0d959ff1d27b29c52abf289a0d3c28054ebe6b2caa6df00d3455c1bf7" exitCode=0
Oct 30 22:13:41.129101 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.129044    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q" event={"ID":"415f6ded-e8b1-4004-8c65-fa9aecaf195b","Type":"ContainerDied","Data":"08a44ff0d959ff1d27b29c52abf289a0d3c28054ebe6b2caa6df00d3455c1bf7"}
Oct 30 22:13:41.278974 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.278945    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 22:13:41.464084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.464051    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415f6ded-e8b1-4004-8c65-fa9aecaf195b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f6ded-e8b1-4004-8c65-fa9aecaf195b-local-certs\") pod \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") "
Oct 30 22:13:41.464267 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.464098    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-dns-cert\") pod \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") "
Oct 30 22:13:41.464267 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.464126    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-cacerts\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-cacerts\") pod \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") "
Oct 30 22:13:41.464425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.464308    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-kubeconfig\") pod \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") "
Oct 30 22:13:41.464425 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.464412    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-ca-configmap\") pod \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") "
Oct 30 22:13:41.464541 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.464476    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-kube-api-access-265qb\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-kube-api-access-265qb\") pod \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") "
Oct 30 22:13:41.464596 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.464575    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-token\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-token\") pod \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\" (UID: \"415f6ded-e8b1-4004-8c65-fa9aecaf195b\") "
Oct 30 22:13:41.464891 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.464859    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-ca-configmap" pod "415f6ded-e8b1-4004-8c65-fa9aecaf195b" (UID: "415f6ded-e8b1-4004-8c65-fa9aecaf195b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:41.465397 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.465372    5886 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-ca-configmap\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:41.467046 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.467017    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-kubeconfig" pod "415f6ded-e8b1-4004-8c65-fa9aecaf195b" (UID: "415f6ded-e8b1-4004-8c65-fa9aecaf195b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:41.467294 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.467270    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415f6ded-e8b1-4004-8c65-fa9aecaf195b-local-certs" pod "415f6ded-e8b1-4004-8c65-fa9aecaf195b" (UID: "415f6ded-e8b1-4004-8c65-fa9aecaf195b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:41.467581 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.467560    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-cacerts" pod "415f6ded-e8b1-4004-8c65-fa9aecaf195b" (UID: "415f6ded-e8b1-4004-8c65-fa9aecaf195b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:41.467870 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.467849    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-kube-api-access-265qb" pod "415f6ded-e8b1-4004-8c65-fa9aecaf195b" (UID: "415f6ded-e8b1-4004-8c65-fa9aecaf195b"). InnerVolumeSpecName "kube-api-access-265qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:41.468237 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.468217    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-token" pod "415f6ded-e8b1-4004-8c65-fa9aecaf195b" (UID: "415f6ded-e8b1-4004-8c65-fa9aecaf195b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:41.468319 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.468298    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-dns-cert" pod "415f6ded-e8b1-4004-8c65-fa9aecaf195b" (UID: "415f6ded-e8b1-4004-8c65-fa9aecaf195b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:41.567151 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.567058    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:41.567151 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.567099    5886 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f6ded-e8b1-4004-8c65-fa9aecaf195b-local-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:41.567151 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.567116    5886 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-csr-dns-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:41.567151 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.567130    5886 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-cacerts\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:41.567151 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.567143    5886 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/415f6ded-e8b1-4004-8c65-fa9aecaf195b-istio-kubeconfig\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:41.567494 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:41.567188    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-265qb\" (UniqueName: \"kubernetes.io/projected/415f6ded-e8b1-4004-8c65-fa9aecaf195b-kube-api-access-265qb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:42.136492 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:42.136460    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"
Oct 30 22:13:42.136492 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:42.136483    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q" event={"ID":"415f6ded-e8b1-4004-8c65-fa9aecaf195b","Type":"ContainerDied","Data":"a79b25754ae6fd75006829ddcf2c34eeb7a5db7e5f5efd3d74fa6738a0d7b325"}
Oct 30 22:13:42.136982 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:42.136544    5886 scope.go:122] "RemoveContainer" containerID="08a44ff0d959ff1d27b29c52abf289a0d3c28054ebe6b2caa6df00d3455c1bf7"
Oct 30 22:13:42.198987 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:42.198949    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"]
Oct 30 22:13:42.210363 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:42.210332    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6cd5q"]
Oct 30 22:13:43.771842 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:43.771808    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"]
Oct 30 22:13:43.991976 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:43.991939    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415f6ded-e8b1-4004-8c65-fa9aecaf195b" path="/var/lib/kubelet/pods/415f6ded-e8b1-4004-8c65-fa9aecaf195b/volumes"
Oct 30 22:13:48.283879 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.283837    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"]
Oct 30 22:13:48.284515 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.284137    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd" podUID="78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" containerName="maas-api" containerID="cri-o://599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89" gracePeriod=30
Oct 30 22:13:48.582218 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.582092    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 22:13:48.672432 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.672391    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-kube-api-access-ljlws\" (UniqueName: \"kubernetes.io/projected/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-kube-api-access-ljlws\") pod \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") "
Oct 30 22:13:48.672641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.672592    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-openshift-service-ca\") pod \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") "
Oct 30 22:13:48.672641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.672620    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-metrics-certs\") pod \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") "
Oct 30 22:13:48.672773 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.672644    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-maas-api-tls\") pod \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\" (UID: \"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a\") "
Oct 30 22:13:48.673012 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.672985    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-openshift-service-ca" pod "78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" (UID: "78d2b79f-b40a-411d-bf1c-9d5a101b8e7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:48.673116 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.673101    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:48.675168 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.675123    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-kube-api-access-ljlws" pod "78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" (UID: "78d2b79f-b40a-411d-bf1c-9d5a101b8e7a"). InnerVolumeSpecName "kube-api-access-ljlws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:48.675308 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.675266    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-maas-api-tls" pod "78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" (UID: "78d2b79f-b40a-411d-bf1c-9d5a101b8e7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:48.675367 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.675297    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-metrics-certs" pod "78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" (UID: "78d2b79f-b40a-411d-bf1c-9d5a101b8e7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:48.705543 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.705178    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"]
Oct 30 22:13:48.705543 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.705450    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b" podUID="e625489f-694d-4ca8-95f6-3192e3e084e4" containerName="maas-api" containerID="cri-o://700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421" gracePeriod=30
Oct 30 22:13:48.773101 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.773060    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k" podUID="c2ceb265-b14e-4cbc-a8a4-9ea026496943" containerName="payload-pre-processing" containerID="cri-o://7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c" gracePeriod=25
Oct 30 22:13:48.774129 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.774099    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:48.774129 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.774131    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:48.774340 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.774145    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljlws\" (UniqueName: \"kubernetes.io/projected/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a-kube-api-access-ljlws\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:48.958905 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:48.958877    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 22:13:49.024978 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.024575    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"]
Oct 30 22:13:49.024978 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.024899    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy" containerID="cri-o://b4e9822ffdba52e9b03cee6c6f595919839aa020bf4570f094f0d34d73122651" gracePeriod=30
Oct 30 22:13:49.027560 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.027533    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"]
Oct 30 22:13:49.027759 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.027738    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy" containerID="cri-o://fce73bd2c701c040500f65fa3d5a8d6322ee0e153ef0bf424bd923d362d5d8a0" gracePeriod=30
Oct 30 22:13:49.077619 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.077534    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-metrics-certs\") pod \"e625489f-694d-4ca8-95f6-3192e3e084e4\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") "
Oct 30 22:13:49.078762 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.077835    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-maas-api-tls\") pod \"e625489f-694d-4ca8-95f6-3192e3e084e4\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") "
Oct 30 22:13:49.079117 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.079095    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e625489f-694d-4ca8-95f6-3192e3e084e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e625489f-694d-4ca8-95f6-3192e3e084e4-openshift-service-ca\") pod \"e625489f-694d-4ca8-95f6-3192e3e084e4\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") "
Oct 30 22:13:49.079328 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.079309    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e625489f-694d-4ca8-95f6-3192e3e084e4-kube-api-access-vff4q\" (UniqueName: \"kubernetes.io/projected/e625489f-694d-4ca8-95f6-3192e3e084e4-kube-api-access-vff4q\") pod \"e625489f-694d-4ca8-95f6-3192e3e084e4\" (UID: \"e625489f-694d-4ca8-95f6-3192e3e084e4\") "
Oct 30 22:13:49.079780 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.079750    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e625489f-694d-4ca8-95f6-3192e3e084e4-openshift-service-ca" pod "e625489f-694d-4ca8-95f6-3192e3e084e4" (UID: "e625489f-694d-4ca8-95f6-3192e3e084e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:49.080071 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.080051    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e625489f-694d-4ca8-95f6-3192e3e084e4-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:49.080581 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.080552    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-metrics-certs" pod "e625489f-694d-4ca8-95f6-3192e3e084e4" (UID: "e625489f-694d-4ca8-95f6-3192e3e084e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:49.080700 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.080680    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-maas-api-tls" pod "e625489f-694d-4ca8-95f6-3192e3e084e4" (UID: "e625489f-694d-4ca8-95f6-3192e3e084e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:49.082701 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.082672    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e625489f-694d-4ca8-95f6-3192e3e084e4-kube-api-access-vff4q" pod "e625489f-694d-4ca8-95f6-3192e3e084e4" (UID: "e625489f-694d-4ca8-95f6-3192e3e084e4"). InnerVolumeSpecName "kube-api-access-vff4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:49.181589 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.181552    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vff4q\" (UniqueName: \"kubernetes.io/projected/e625489f-694d-4ca8-95f6-3192e3e084e4-kube-api-access-vff4q\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:49.181589 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.181591    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:49.181812 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.181605    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e625489f-694d-4ca8-95f6-3192e3e084e4-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:49.188739 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.188698    5886 generic.go:356] "Generic (PLEG): container finished" podID="e625489f-694d-4ca8-95f6-3192e3e084e4" containerID="700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421" exitCode=0
Oct 30 22:13:49.188913 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.188751    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"
Oct 30 22:13:49.188913 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.188788    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b" event={"ID":"e625489f-694d-4ca8-95f6-3192e3e084e4","Type":"ContainerDied","Data":"700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421"}
Oct 30 22:13:49.188913 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.188846    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b" event={"ID":"e625489f-694d-4ca8-95f6-3192e3e084e4","Type":"ContainerDied","Data":"21519bf0d023d24be2f454aaea98de153864a460d855d57b041af68a7c57c6c5"}
Oct 30 22:13:49.188913 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.188879    5886 scope.go:122] "RemoveContainer" containerID="700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421"
Oct 30 22:13:49.190571 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.190544    5886 generic.go:356] "Generic (PLEG): container finished" podID="78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" containerID="599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89" exitCode=0
Oct 30 22:13:49.190723 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.190602    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"
Oct 30 22:13:49.190723 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.190658    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd" event={"ID":"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a","Type":"ContainerDied","Data":"599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89"}
Oct 30 22:13:49.190906 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.190731    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd" event={"ID":"78d2b79f-b40a-411d-bf1c-9d5a101b8e7a","Type":"ContainerDied","Data":"f79878b397c8757853704acb362bac09dd458374ea46c82711922950d4bad123"}
Oct 30 22:13:49.202481 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.202457    5886 scope.go:122] "RemoveContainer" containerID="700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421"
Oct 30 22:13:49.202978 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:49.202948    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421\": container with ID starting with 700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421 not found: ID does not exist" containerID="700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421"
Oct 30 22:13:49.203105 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.202988    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421"} err="failed to get container status \"700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421\": rpc error: code = NotFound desc = could not find container \"700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421\": container with ID starting with 700031c83acf79d983c01c87a74f376fe6f5bb1324e40df04090250677128421 not found: ID does not exist"
Oct 30 22:13:49.203105 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.203010    5886 scope.go:122] "RemoveContainer" containerID="599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89"
Oct 30 22:13:49.218775 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.218741    5886 scope.go:122] "RemoveContainer" containerID="599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89"
Oct 30 22:13:49.219059 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:13:49.219042    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89\": container with ID starting with 599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89 not found: ID does not exist" containerID="599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89"
Oct 30 22:13:49.219221 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.219065    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89"} err="failed to get container status \"599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89\": rpc error: code = NotFound desc = could not find container \"599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89\": container with ID starting with 599d26bb6fba65bdbdbcb7bf0bad2145e3967c9872af9b89fd162e4e7b6e8f89 not found: ID does not exist"
Oct 30 22:13:49.258484 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.258451    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"]
Oct 30 22:13:49.261687 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.261652    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-805edc-67cd6cb7d9-gkp9b"]
Oct 30 22:13:49.306080 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.306038    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"]
Oct 30 22:13:49.329216 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.321189    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5d3601-6d8f979fb5-9xjtd"]
Oct 30 22:13:49.989002 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.988958    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" path="/var/lib/kubelet/pods/78d2b79f-b40a-411d-bf1c-9d5a101b8e7a/volumes"
Oct 30 22:13:49.989567 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:49.989546    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e625489f-694d-4ca8-95f6-3192e3e084e4" path="/var/lib/kubelet/pods/e625489f-694d-4ca8-95f6-3192e3e084e4/volumes"
Oct 30 22:13:51.002047 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:51.001995    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k" podUID="c2ceb265-b14e-4cbc-a8a4-9ea026496943" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:13:54.227742 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.227604    5886 generic.go:356] "Generic (PLEG): container finished" podID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerID="b4e9822ffdba52e9b03cee6c6f595919839aa020bf4570f094f0d34d73122651" exitCode=0
Oct 30 22:13:54.227742 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.227648    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" event={"ID":"bbf54c5f-a5f8-421a-be0e-c75ee4906734","Type":"ContainerDied","Data":"b4e9822ffdba52e9b03cee6c6f595919839aa020bf4570f094f0d34d73122651"}
Oct 30 22:13:54.230267 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.230239    5886 generic.go:356] "Generic (PLEG): container finished" podID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerID="fce73bd2c701c040500f65fa3d5a8d6322ee0e153ef0bf424bd923d362d5d8a0" exitCode=0
Oct 30 22:13:54.230376 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.230320    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" event={"ID":"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a","Type":"ContainerDied","Data":"fce73bd2c701c040500f65fa3d5a8d6322ee0e153ef0bf424bd923d362d5d8a0"}
Oct 30 22:13:54.312262 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.312233    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 22:13:54.354456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.354426    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 22:13:54.439860 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.439826    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-certs\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440066 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.439877    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-kube-api-access-x9cdp\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-kube-api-access-x9cdp\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.440066 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.439910    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-envoy\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.440066 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.439947    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-data\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440066 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.439988    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-envoy\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440066 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440019    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-credential-socket\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.440066 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440062    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istiod-ca-cert\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440448 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440095    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-socket\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.440448 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440135    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-certs" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.440448 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440145    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-token\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-token\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.440448 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440231    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-socket\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440448 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440270    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-credential-socket\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440448 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440306    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-podinfo\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.440448 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440351    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-certs\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.440448 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440448    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-token\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-token\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440868 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440532    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istiod-ca-cert\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.440868 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440584    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-podinfo\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440868 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440591    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-socket" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.440868 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440643    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-kube-api-access-nf4rj\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-kube-api-access-nf4rj\") pod \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\" (UID: \"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a\") "
Oct 30 22:13:54.440868 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.440709    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-data\") pod \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\" (UID: \"bbf54c5f-a5f8-421a-be0e-c75ee4906734\") "
Oct 30 22:13:54.441278 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.441260    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.441349 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.441282    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.443292 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.443263    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-credential-socket" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.443458 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.443414    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-socket" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.443458 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.443424    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-envoy" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.443587 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.443484    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istiod-ca-cert" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:54.443675 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.443652    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-data" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.443833 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.443807    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-token" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:54.443833 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.443824    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-credential-socket" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.443964 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.443859    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-kube-api-access-x9cdp" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "kube-api-access-x9cdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:54.444100 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.444080    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-data" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.444206 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.444119    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istiod-ca-cert" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:54.444383 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.444361    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-certs" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.445499 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.445477    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-token" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:54.445866 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.445850    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-podinfo" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:54.446393 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.446376    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-kube-api-access-nf4rj" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "kube-api-access-nf4rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:54.446755 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.446735    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-envoy" pod "bbf54c5f-a5f8-421a-be0e-c75ee4906734" (UID: "bbf54c5f-a5f8-421a-be0e-c75ee4906734"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:54.448236 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.448212    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-podinfo" pod "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" (UID: "4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:54.542862 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542827    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.542862 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542857    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542868    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542891    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4rj\" (UniqueName: \"kubernetes.io/projected/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-kube-api-access-nf4rj\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542901    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542911    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9cdp\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-kube-api-access-x9cdp\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542919    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542927    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542935    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542942    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542950    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542958    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542965    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542974    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542982    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf54c5f-a5f8-421a-be0e-c75ee4906734-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:54.543058 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:54.542990    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf54c5f-a5f8-421a-be0e-c75ee4906734-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:13:55.237560 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.237526    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"
Oct 30 22:13:55.237560 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.237537    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz" event={"ID":"bbf54c5f-a5f8-421a-be0e-c75ee4906734","Type":"ContainerDied","Data":"273bc98dc34fecacb2f923e57ba489b6d4eae48070e0a1c8245013334cecebe3"}
Oct 30 22:13:55.238148 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.237603    5886 scope.go:122] "RemoveContainer" containerID="b4e9822ffdba52e9b03cee6c6f595919839aa020bf4570f094f0d34d73122651"
Oct 30 22:13:55.239044 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.239003    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8" event={"ID":"4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a","Type":"ContainerDied","Data":"17eec25294ec679f12ddbcf551f1e4bef269fa5222d28881f656a6cb0dfd61f5"}
Oct 30 22:13:55.239044 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.239041    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"
Oct 30 22:13:55.248687 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.248577    5886 scope.go:122] "RemoveContainer" containerID="fce73bd2c701c040500f65fa3d5a8d6322ee0e153ef0bf424bd923d362d5d8a0"
Oct 30 22:13:55.305148 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.305109    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"]
Oct 30 22:13:55.307952 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.307926    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7bsl8"]
Oct 30 22:13:55.351186 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.351096    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"]
Oct 30 22:13:55.357248 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.357214    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4rjnz"]
Oct 30 22:13:55.987877 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.987841    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" path="/var/lib/kubelet/pods/4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a/volumes"
Oct 30 22:13:55.988367 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:13:55.988351    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" path="/var/lib/kubelet/pods/bbf54c5f-a5f8-421a-be0e-c75ee4906734/volumes"
Oct 30 22:14:01.001850 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:01.001811    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k" podUID="c2ceb265-b14e-4cbc-a8a4-9ea026496943" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:03.056001 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:03.055963    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"]
Oct 30 22:14:03.056440 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:03.056183    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy" containerID="cri-o://df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8" gracePeriod=30
Oct 30 22:14:04.518241 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:04.518205    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"]
Oct 30 22:14:04.518692 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:04.518419    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy" containerID="cri-o://de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6" gracePeriod=30
Oct 30 22:14:08.324924 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.324855    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 22:14:08.337620 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.337587    5886 generic.go:356] "Generic (PLEG): container finished" podID="c1254d30-70fd-4247-b20b-18233b6a6992" containerID="df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8" exitCode=0
Oct 30 22:14:08.337791 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.337651    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"
Oct 30 22:14:08.337791 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.337673    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" event={"ID":"c1254d30-70fd-4247-b20b-18233b6a6992","Type":"ContainerDied","Data":"df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8"}
Oct 30 22:14:08.337791 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.337731    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt" event={"ID":"c1254d30-70fd-4247-b20b-18233b6a6992","Type":"ContainerDied","Data":"159037368e78107613e8561936c59a9317e0a14e503256ffd8df9cc6a7e63092"}
Oct 30 22:14:08.337791 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.337755    5886 scope.go:122] "RemoveContainer" containerID="df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8"
Oct 30 22:14:08.347999 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.347974    5886 scope.go:122] "RemoveContainer" containerID="df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8"
Oct 30 22:14:08.348319 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:08.348293    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8\": container with ID starting with df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8 not found: ID does not exist" containerID="df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8"
Oct 30 22:14:08.348421 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.348325    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8"} err="failed to get container status \"df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8\": rpc error: code = NotFound desc = could not find container \"df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8\": container with ID starting with df8f78fd0587ed52710e19f8c3c9b313575147720bc6d2596892b00b8f3aa6f8 not found: ID does not exist"
Oct 30 22:14:08.415785 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.415745    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-kube-api-access-srlkf\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-kube-api-access-srlkf\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.416084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.415802    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1254d30-70fd-4247-b20b-18233b6a6992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1254d30-70fd-4247-b20b-18233b6a6992-istio-podinfo\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.416084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.415942    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-socket\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.416084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.415975    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-credential-socket\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.416084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.416021    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-envoy\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.416084 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.416081    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-certs\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.416999 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.416112    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-istio-token\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-istio-token\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.416999 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.416190    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-data\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.416999 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.416224    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1254d30-70fd-4247-b20b-18233b6a6992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1254d30-70fd-4247-b20b-18233b6a6992-istiod-ca-cert\") pod \"c1254d30-70fd-4247-b20b-18233b6a6992\" (UID: \"c1254d30-70fd-4247-b20b-18233b6a6992\") "
Oct 30 22:14:08.417205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.417004    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1254d30-70fd-4247-b20b-18233b6a6992-istiod-ca-cert" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:08.418036 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.417991    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-certs" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:08.418186 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.418138    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-data" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:08.418304 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.418226    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-credential-socket" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:08.418502 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.418479    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-socket" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:08.420732 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.420710    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-envoy" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:08.420943 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.420924    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1254d30-70fd-4247-b20b-18233b6a6992-istio-podinfo" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:08.421037 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.420996    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-istio-token" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:08.434332 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.434277    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-kube-api-access-srlkf" pod "c1254d30-70fd-4247-b20b-18233b6a6992" (UID: "c1254d30-70fd-4247-b20b-18233b6a6992"). InnerVolumeSpecName "kube-api-access-srlkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:08.517970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.517931    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.517970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.517961    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.517970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.517972    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.518299 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.517982    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.518299 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.517990    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1254d30-70fd-4247-b20b-18233b6a6992-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.518299 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.517999    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srlkf\" (UniqueName: \"kubernetes.io/projected/c1254d30-70fd-4247-b20b-18233b6a6992-kube-api-access-srlkf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.518299 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.518008    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1254d30-70fd-4247-b20b-18233b6a6992-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.518299 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.518016    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.518299 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.518025    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1254d30-70fd-4247-b20b-18233b6a6992-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:08.685265 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.685226    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"]
Oct 30 22:14:08.696194 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:08.696145    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ea40ee-openshift-default-9bd9b546f-5jvnt"]
Oct 30 22:14:09.037987 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.037959    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 22:14:09.126433 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.126401    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ceb265-b14e-4cbc-a8a4-9ea026496943-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ceb265-b14e-4cbc-a8a4-9ea026496943-plugins-config-volume\") pod \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\" (UID: \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\") "
Oct 30 22:14:09.126646 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.126607    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ceb265-b14e-4cbc-a8a4-9ea026496943-kube-api-access-glq4g\" (UniqueName: \"kubernetes.io/projected/c2ceb265-b14e-4cbc-a8a4-9ea026496943-kube-api-access-glq4g\") pod \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\" (UID: \"c2ceb265-b14e-4cbc-a8a4-9ea026496943\") "
Oct 30 22:14:09.126861 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.126831    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ceb265-b14e-4cbc-a8a4-9ea026496943-plugins-config-volume" pod "c2ceb265-b14e-4cbc-a8a4-9ea026496943" (UID: "c2ceb265-b14e-4cbc-a8a4-9ea026496943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:09.126999 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.126984    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ceb265-b14e-4cbc-a8a4-9ea026496943-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.128685 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.128661    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ceb265-b14e-4cbc-a8a4-9ea026496943-kube-api-access-glq4g" pod "c2ceb265-b14e-4cbc-a8a4-9ea026496943" (UID: "c2ceb265-b14e-4cbc-a8a4-9ea026496943"). InnerVolumeSpecName "kube-api-access-glq4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:09.228674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.228589    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glq4g\" (UniqueName: \"kubernetes.io/projected/c2ceb265-b14e-4cbc-a8a4-9ea026496943-kube-api-access-glq4g\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.346651 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.346610    5886 generic.go:356] "Generic (PLEG): container finished" podID="c2ceb265-b14e-4cbc-a8a4-9ea026496943" containerID="7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c" exitCode=0
Oct 30 22:14:09.347111 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.346662    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"
Oct 30 22:14:09.347111 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.346688    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k" event={"ID":"c2ceb265-b14e-4cbc-a8a4-9ea026496943","Type":"ContainerDied","Data":"7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c"}
Oct 30 22:14:09.347111 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.346743    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k" event={"ID":"c2ceb265-b14e-4cbc-a8a4-9ea026496943","Type":"ContainerDied","Data":"549fe5a76e6f4ae28210e1da8b680ff797f8a3db3c069c12d8ac100a09158fd6"}
Oct 30 22:14:09.347111 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.346765    5886 scope.go:122] "RemoveContainer" containerID="7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c"
Oct 30 22:14:09.357342 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.357309    5886 scope.go:122] "RemoveContainer" containerID="7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c"
Oct 30 22:14:09.357588 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:09.357570    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c\": container with ID starting with 7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c not found: ID does not exist" containerID="7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c"
Oct 30 22:14:09.357663 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.357595    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c"} err="failed to get container status \"7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c\": rpc error: code = NotFound desc = could not find container \"7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c\": container with ID starting with 7fa9908b7c0cf907d5dd28978c4c1db55da40b6e246c9c185f752cf2b6cfcc2c not found: ID does not exist"
Oct 30 22:14:09.395909 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.395880    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"]
Oct 30 22:14:09.406926 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.406896    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-805edc-84df88c6c9-wdr4k"]
Oct 30 22:14:09.796036 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.795970    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 22:14:09.835201 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835168    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-podinfo\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835383 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835257    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-kube-api-access-dr96c\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-kube-api-access-dr96c\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835383 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835283    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-certs\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835383 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835335    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-socket\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835383 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835377    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-token\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-token\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835483    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-data\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835519    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-credential-socket\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835549    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-envoy\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835602    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istiod-ca-cert\") pod \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\" (UID: \"970e4f4a-4008-4139-a720-0c6d8e1bf0b7\") "
Oct 30 22:14:09.835674 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835617    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-certs" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:09.835941 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835754    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-socket" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:09.835941 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.835866    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-credential-socket" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:09.836079 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.836044    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-data" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:09.836217 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.836198    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.836298 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.836220    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.836298 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.836234    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.836298 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.836247    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.836474 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.836392    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istiod-ca-cert" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:09.837631 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.837592    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-kube-api-access-dr96c" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "kube-api-access-dr96c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:09.837804 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.837784    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-podinfo" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:09.838038 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.838012    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-envoy" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:09.838291 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.838271    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-token" pod "970e4f4a-4008-4139-a720-0c6d8e1bf0b7" (UID: "970e4f4a-4008-4139-a720-0c6d8e1bf0b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:09.940075 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.940017    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr96c\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-kube-api-access-dr96c\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.940075 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.940058    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.940075 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.940085    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.940571 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.940099    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.940571 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.940115    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e4f4a-4008-4139-a720-0c6d8e1bf0b7-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:09.988919 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.988882    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" path="/var/lib/kubelet/pods/c1254d30-70fd-4247-b20b-18233b6a6992/volumes"
Oct 30 22:14:09.989300 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:09.989283    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ceb265-b14e-4cbc-a8a4-9ea026496943" path="/var/lib/kubelet/pods/c2ceb265-b14e-4cbc-a8a4-9ea026496943/volumes"
Oct 30 22:14:10.353970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.353874    5886 generic.go:356] "Generic (PLEG): container finished" podID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerID="de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6" exitCode=0
Oct 30 22:14:10.353970 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.353943    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"
Oct 30 22:14:10.354527 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.353965    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" event={"ID":"970e4f4a-4008-4139-a720-0c6d8e1bf0b7","Type":"ContainerDied","Data":"de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6"}
Oct 30 22:14:10.354527 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.354031    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs" event={"ID":"970e4f4a-4008-4139-a720-0c6d8e1bf0b7","Type":"ContainerDied","Data":"d8e5b7c6b6a64fc020683a1c5391e0a047ef741045b69260a125f3dd06a32a3a"}
Oct 30 22:14:10.354527 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.354065    5886 scope.go:122] "RemoveContainer" containerID="de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6"
Oct 30 22:14:10.363700 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.363679    5886 scope.go:122] "RemoveContainer" containerID="de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6"
Oct 30 22:14:10.363995 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:10.363970    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6\": container with ID starting with de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6 not found: ID does not exist" containerID="de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6"
Oct 30 22:14:10.364103 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.364003    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6"} err="failed to get container status \"de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6\": rpc error: code = NotFound desc = could not find container \"de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6\": container with ID starting with de78e9f162cd2aded5a22b8bedce9a88c14a780033817fa3c5f4ff6c3748dfe6 not found: ID does not exist"
Oct 30 22:14:10.418086 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.418052    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"]
Oct 30 22:14:10.421168 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:10.421135    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-09eabe-openshift-default-6d877fd87c-4n5zs"]
Oct 30 22:14:11.031514 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:11.031475    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:14:11.988733 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:11.988697    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" path="/var/lib/kubelet/pods/970e4f4a-4008-4139-a720-0c6d8e1bf0b7/volumes"
Oct 30 22:14:20.331847 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:20.331810    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"]
Oct 30 22:14:20.332550 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:20.332084    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy" containerID="cri-o://1a76c3916e102a532acf30cf81aa9f9992befde7fd6bff3810f474e7f223cfc4" gracePeriod=30
Oct 30 22:14:25.472593 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.472102    5886 generic.go:356] "Generic (PLEG): container finished" podID="4631b858-9891-4eda-8a4a-8142682819b3" containerID="1a76c3916e102a532acf30cf81aa9f9992befde7fd6bff3810f474e7f223cfc4" exitCode=0
Oct 30 22:14:25.472593 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.472226    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" event={"ID":"4631b858-9891-4eda-8a4a-8142682819b3","Type":"ContainerDied","Data":"1a76c3916e102a532acf30cf81aa9f9992befde7fd6bff3810f474e7f223cfc4"}
Oct 30 22:14:25.472593 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.472359    5886 scope.go:122] "RemoveContainer" containerID="b4d47d01eb1e6b446d45f641c0d1f9917acc54f8e36e8638e45fc934add0b906"
Oct 30 22:14:25.473623 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.473544    5886 patch_prober.go:29] interesting pod/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:14:25.473623 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.473598    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:14:25.599742 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.599712    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 22:14:25.716603 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716569    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-kube-api-access-tt9m4\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-kube-api-access-tt9m4\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.716807 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716613    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-certs\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.716807 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716669    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-socket\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.716807 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716713    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4631b858-9891-4eda-8a4a-8142682819b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4631b858-9891-4eda-8a4a-8142682819b3-istiod-ca-cert\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.716807 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716767    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4631b858-9891-4eda-8a4a-8142682819b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4631b858-9891-4eda-8a4a-8142682819b3-istio-podinfo\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.717041 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716809    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-data\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.717041 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716853    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-istio-token\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-istio-token\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.717041 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716894    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-envoy\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.717041 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.716949    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-credential-socket\") pod \"4631b858-9891-4eda-8a4a-8142682819b3\" (UID: \"4631b858-9891-4eda-8a4a-8142682819b3\") "
Oct 30 22:14:25.717041 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.717015    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-certs" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.717041 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.717026    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-socket" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.717404 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.717365    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.717404 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.717388    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.717644 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.717623    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-credential-socket" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.717737 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.717637    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-data" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.717852 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.717833    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631b858-9891-4eda-8a4a-8142682819b3-istiod-ca-cert" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:25.719411 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.719387    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4631b858-9891-4eda-8a4a-8142682819b3-istio-podinfo" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:25.719411 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.719393    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-kube-api-access-tt9m4" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "kube-api-access-tt9m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:25.719993 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.719975    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-istio-token" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:25.720287 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.720272    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-envoy" pod "4631b858-9891-4eda-8a4a-8142682819b3" (UID: "4631b858-9891-4eda-8a4a-8142682819b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:25.819065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.819022    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.819065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.819055    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.819065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.819068    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.819360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.819082    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4631b858-9891-4eda-8a4a-8142682819b3-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.819360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.819094    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt9m4\" (UniqueName: \"kubernetes.io/projected/4631b858-9891-4eda-8a4a-8142682819b3-kube-api-access-tt9m4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.819360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.819107    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4631b858-9891-4eda-8a4a-8142682819b3-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:25.819360 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:25.819118    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4631b858-9891-4eda-8a4a-8142682819b3-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:26.479439 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:26.479400    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx" event={"ID":"4631b858-9891-4eda-8a4a-8142682819b3","Type":"ContainerDied","Data":"9f350678a66617d385f58dc14dba2707fa7e9ebb6ec8837ca1aebc9a5c0118e1"}
Oct 30 22:14:26.479439 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:26.479435    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"
Oct 30 22:14:26.479950 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:26.479461    5886 scope.go:122] "RemoveContainer" containerID="1a76c3916e102a532acf30cf81aa9f9992befde7fd6bff3810f474e7f223cfc4"
Oct 30 22:14:26.546234 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:26.546194    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"]
Oct 30 22:14:26.557513 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:26.557483    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ed0866-openshift-default-54994c7bf7-6stqx"]
Oct 30 22:14:27.988187 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:27.988118    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4631b858-9891-4eda-8a4a-8142682819b3" path="/var/lib/kubelet/pods/4631b858-9891-4eda-8a4a-8142682819b3/volumes"
Oct 30 22:14:31.634558 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:31.634515    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"]
Oct 30 22:14:33.564078 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.564040    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"]
Oct 30 22:14:33.564613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.564283    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw" podUID="2811cac3-c5ae-4ae5-988b-c4839e9c8364" containerName="maas-api" containerID="cri-o://51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8" gracePeriod=30
Oct 30 22:14:33.847031 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.847006    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 22:14:33.917902 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.917871    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls\") pod \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") "
Oct 30 22:14:33.918091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.917911    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-metrics-certs\") pod \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") "
Oct 30 22:14:33.918091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.918028    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2811cac3-c5ae-4ae5-988b-c4839e9c8364-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2811cac3-c5ae-4ae5-988b-c4839e9c8364-openshift-service-ca\") pod \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") "
Oct 30 22:14:33.918249 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.918091    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2811cac3-c5ae-4ae5-988b-c4839e9c8364-kube-api-access-cj5zf\" (UniqueName: \"kubernetes.io/projected/2811cac3-c5ae-4ae5-988b-c4839e9c8364-kube-api-access-cj5zf\") pod \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\" (UID: \"2811cac3-c5ae-4ae5-988b-c4839e9c8364\") "
Oct 30 22:14:33.918709 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.918673    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2811cac3-c5ae-4ae5-988b-c4839e9c8364-openshift-service-ca" pod "2811cac3-c5ae-4ae5-988b-c4839e9c8364" (UID: "2811cac3-c5ae-4ae5-988b-c4839e9c8364"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:33.920706 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.920672    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-metrics-certs" pod "2811cac3-c5ae-4ae5-988b-c4839e9c8364" (UID: "2811cac3-c5ae-4ae5-988b-c4839e9c8364"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:33.920903 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.920876    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls" pod "2811cac3-c5ae-4ae5-988b-c4839e9c8364" (UID: "2811cac3-c5ae-4ae5-988b-c4839e9c8364"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:33.923308 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:33.923285    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2811cac3-c5ae-4ae5-988b-c4839e9c8364-kube-api-access-cj5zf" pod "2811cac3-c5ae-4ae5-988b-c4839e9c8364" (UID: "2811cac3-c5ae-4ae5-988b-c4839e9c8364"). InnerVolumeSpecName "kube-api-access-cj5zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:34.020142 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.020111    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2811cac3-c5ae-4ae5-988b-c4839e9c8364-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:34.020142 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.020138    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj5zf\" (UniqueName: \"kubernetes.io/projected/2811cac3-c5ae-4ae5-988b-c4839e9c8364-kube-api-access-cj5zf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:34.020420 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.020149    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:34.020420 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.020177    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2811cac3-c5ae-4ae5-988b-c4839e9c8364-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:34.537221 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.537178    5886 generic.go:356] "Generic (PLEG): container finished" podID="2811cac3-c5ae-4ae5-988b-c4839e9c8364" containerID="51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8" exitCode=0
Oct 30 22:14:34.537417 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.537224    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"
Oct 30 22:14:34.537417 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.537247    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw" event={"ID":"2811cac3-c5ae-4ae5-988b-c4839e9c8364","Type":"ContainerDied","Data":"51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8"}
Oct 30 22:14:34.537417 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.537302    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw" event={"ID":"2811cac3-c5ae-4ae5-988b-c4839e9c8364","Type":"ContainerDied","Data":"b9cade5b99db41b0ea1d33d47cfbe6cb950e6c8d48a6e80ca555809c8a399751"}
Oct 30 22:14:34.537417 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.537325    5886 scope.go:122] "RemoveContainer" containerID="51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8"
Oct 30 22:14:34.546762 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.546744    5886 scope.go:122] "RemoveContainer" containerID="51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8"
Oct 30 22:14:34.547016 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:34.546996    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8\": container with ID starting with 51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8 not found: ID does not exist" containerID="51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8"
Oct 30 22:14:34.547091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.547025    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8"} err="failed to get container status \"51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8\": rpc error: code = NotFound desc = could not find container \"51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8\": container with ID starting with 51fa2962bdb8f739545073cc0a9783cff43b95f9aa6dd56da06fffd975cdbed8 not found: ID does not exist"
Oct 30 22:14:34.588457 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.588355    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"]
Oct 30 22:14:34.598309 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:34.598276    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-5494e8-7698574f87-hjxnw"]
Oct 30 22:14:35.988541 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:35.988501    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2811cac3-c5ae-4ae5-988b-c4839e9c8364" path="/var/lib/kubelet/pods/2811cac3-c5ae-4ae5-988b-c4839e9c8364/volumes"
Oct 30 22:14:36.635261 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:36.635195    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc" podUID="1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" containerName="payload-processing" containerID="cri-o://4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a" gracePeriod=25
Oct 30 22:14:41.027993 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:41.027910    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:14:42.786669 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:42.786632    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"]
Oct 30 22:14:42.787485 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:42.787451    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66" podUID="7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" containerName="maas-api" containerID="cri-o://c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680" gracePeriod=30
Oct 30 22:14:43.056631 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.056555    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 22:14:43.226730 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.226585    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-metrics-certs\") pod \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") "
Oct 30 22:14:43.226730 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.226674    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-maas-api-tls\") pod \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") "
Oct 30 22:14:43.227001 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.226786    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-openshift-service-ca\") pod \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") "
Oct 30 22:14:43.227001 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.226852    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-kube-api-access-fml9t\" (UniqueName: \"kubernetes.io/projected/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-kube-api-access-fml9t\") pod \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\" (UID: \"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8\") "
Oct 30 22:14:43.227390 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.227359    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-openshift-service-ca" pod "7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" (UID: "7ed962bd-2ed6-44b4-a4e3-bef92fc054b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:43.228255 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.228214    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:43.229204 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.229144    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-metrics-certs" pod "7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" (UID: "7ed962bd-2ed6-44b4-a4e3-bef92fc054b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:43.229310 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.229238    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-maas-api-tls" pod "7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" (UID: "7ed962bd-2ed6-44b4-a4e3-bef92fc054b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:43.229464 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.229446    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-kube-api-access-fml9t" pod "7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" (UID: "7ed962bd-2ed6-44b4-a4e3-bef92fc054b8"). InnerVolumeSpecName "kube-api-access-fml9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:43.329152 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.329055    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fml9t\" (UniqueName: \"kubernetes.io/projected/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-kube-api-access-fml9t\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:43.329152 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.329106    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:43.329152 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.329121    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:43.604887 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.604802    5886 generic.go:356] "Generic (PLEG): container finished" podID="7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" containerID="c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680" exitCode=0
Oct 30 22:14:43.604887 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.604852    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"
Oct 30 22:14:43.605091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.604857    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66" event={"ID":"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8","Type":"ContainerDied","Data":"c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680"}
Oct 30 22:14:43.605091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.604964    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66" event={"ID":"7ed962bd-2ed6-44b4-a4e3-bef92fc054b8","Type":"ContainerDied","Data":"a82b0addefc1ed6bb48742a639ef799c7b584566d7a3598b809b85df87fb304b"}
Oct 30 22:14:43.605091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.604985    5886 scope.go:122] "RemoveContainer" containerID="c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680"
Oct 30 22:14:43.615225 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.615207    5886 scope.go:122] "RemoveContainer" containerID="c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680"
Oct 30 22:14:43.615511 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:43.615492    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680\": container with ID starting with c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680 not found: ID does not exist" containerID="c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680"
Oct 30 22:14:43.615577 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.615520    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680"} err="failed to get container status \"c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680\": rpc error: code = NotFound desc = could not find container \"c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680\": container with ID starting with c3aeaef0a347e4a8670aaeff7484fd5f14102cd31f8d0d77759334ecb3f50680 not found: ID does not exist"
Oct 30 22:14:43.657670 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.657635    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"]
Oct 30 22:14:43.668740 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.668709    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e06b90-79cd55bf44-5lb66"]
Oct 30 22:14:43.987030 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:43.986994    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" path="/var/lib/kubelet/pods/7ed962bd-2ed6-44b4-a4e3-bef92fc054b8/volumes"
Oct 30 22:14:44.280878 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:44.280779    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc" podUID="1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:52.278409 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:52.278365    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"]
Oct 30 22:14:54.281403 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:54.281357    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc" podUID="1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:14:56.906834 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:56.906805    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 22:14:56.962941 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:56.962910    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-plugins-config-volume\") pod \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\" (UID: \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\") "
Oct 30 22:14:56.963121 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:56.962990    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-kube-api-access-2qtzz\" (UniqueName: \"kubernetes.io/projected/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-kube-api-access-2qtzz\") pod \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\" (UID: \"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2\") "
Oct 30 22:14:56.963365 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:56.963335    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-plugins-config-volume" pod "1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" (UID: "1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:56.963575 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:56.963448    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:56.965232 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:56.965200    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-kube-api-access-2qtzz" pod "1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" (UID: "1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2"). InnerVolumeSpecName "kube-api-access-2qtzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:57.064501 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.064461    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtzz\" (UniqueName: \"kubernetes.io/projected/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2-kube-api-access-2qtzz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:57.279089 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.278988    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh" podUID="e9cef9fb-a937-4586-ad66-b62771a6d36a" containerName="payload-processing" containerID="cri-o://f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6" gracePeriod=25
Oct 30 22:14:57.527175 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.527126    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 22:14:57.569930 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.569850    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cef9fb-a937-4586-ad66-b62771a6d36a-kube-api-access-js2lz\" (UniqueName: \"kubernetes.io/projected/e9cef9fb-a937-4586-ad66-b62771a6d36a-kube-api-access-js2lz\") pod \"e9cef9fb-a937-4586-ad66-b62771a6d36a\" (UID: \"e9cef9fb-a937-4586-ad66-b62771a6d36a\") "
Oct 30 22:14:57.569930 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.569897    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cef9fb-a937-4586-ad66-b62771a6d36a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cef9fb-a937-4586-ad66-b62771a6d36a-plugins-config-volume\") pod \"e9cef9fb-a937-4586-ad66-b62771a6d36a\" (UID: \"e9cef9fb-a937-4586-ad66-b62771a6d36a\") "
Oct 30 22:14:57.570402 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.570373    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cef9fb-a937-4586-ad66-b62771a6d36a-plugins-config-volume" pod "e9cef9fb-a937-4586-ad66-b62771a6d36a" (UID: "e9cef9fb-a937-4586-ad66-b62771a6d36a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:57.572595 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.572564    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cef9fb-a937-4586-ad66-b62771a6d36a-kube-api-access-js2lz" pod "e9cef9fb-a937-4586-ad66-b62771a6d36a" (UID: "e9cef9fb-a937-4586-ad66-b62771a6d36a"). InnerVolumeSpecName "kube-api-access-js2lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:57.671127 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.671095    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2lz\" (UniqueName: \"kubernetes.io/projected/e9cef9fb-a937-4586-ad66-b62771a6d36a-kube-api-access-js2lz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:57.671127 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.671122    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9cef9fb-a937-4586-ad66-b62771a6d36a-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:14:57.692947 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.692910    5886 generic.go:356] "Generic (PLEG): container finished" podID="e9cef9fb-a937-4586-ad66-b62771a6d36a" containerID="f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6" exitCode=0
Oct 30 22:14:57.693116 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.692967    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"
Oct 30 22:14:57.693116 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.693004    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh" event={"ID":"e9cef9fb-a937-4586-ad66-b62771a6d36a","Type":"ContainerDied","Data":"f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6"}
Oct 30 22:14:57.693116 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.693070    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh" event={"ID":"e9cef9fb-a937-4586-ad66-b62771a6d36a","Type":"ContainerDied","Data":"4fc7e7367d68e060c3b6bd069dba9dff36ecead49acfea51cfd573991c1ae78d"}
Oct 30 22:14:57.693116 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.693101    5886 scope.go:122] "RemoveContainer" containerID="f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6"
Oct 30 22:14:57.695838 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.695794    5886 generic.go:356] "Generic (PLEG): container finished" podID="1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" containerID="4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a" exitCode=0
Oct 30 22:14:57.696034 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.695954    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc" event={"ID":"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2","Type":"ContainerDied","Data":"4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a"}
Oct 30 22:14:57.696034 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.696004    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc" event={"ID":"1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2","Type":"ContainerDied","Data":"90165df685e47dc96c4e43b8211caf55fc4b5c6dacbeb48aaca036e6aedff7a8"}
Oct 30 22:14:57.696878 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.696074    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"
Oct 30 22:14:57.716015 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.712108    5886 scope.go:122] "RemoveContainer" containerID="f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6"
Oct 30 22:14:57.719671 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:57.718663    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6\": container with ID starting with f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6 not found: ID does not exist" containerID="f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6"
Oct 30 22:14:57.719671 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.718696    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6"} err="failed to get container status \"f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6\": rpc error: code = NotFound desc = could not find container \"f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6\": container with ID starting with f40fb55fc5ad0398ea566e0545cf9972b2e8bed9908403d28c4a339ac3e482a6 not found: ID does not exist"
Oct 30 22:14:57.719671 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.718718    5886 scope.go:122] "RemoveContainer" containerID="4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a"
Oct 30 22:14:57.732589 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.732564    5886 scope.go:122] "RemoveContainer" containerID="4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a"
Oct 30 22:14:57.732969 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:14:57.732943    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a\": container with ID starting with 4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a not found: ID does not exist" containerID="4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a"
Oct 30 22:14:57.733087 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.732979    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a"} err="failed to get container status \"4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a\": rpc error: code = NotFound desc = could not find container \"4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a\": container with ID starting with 4262b056c37aea355c703faf3dbc1d4df1b48a79517bdbee3a00959dd92c026a not found: ID does not exist"
Oct 30 22:14:57.761785 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.761754    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"]
Oct 30 22:14:57.767382 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.767353    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e06b90-6cfc69b969-mgfkc"]
Oct 30 22:14:57.800751 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.800715    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"]
Oct 30 22:14:57.802872 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.802844    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-ae6492-f69c69bfc-gz2sh"]
Oct 30 22:14:57.989400 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.989363    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" path="/var/lib/kubelet/pods/1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2/volumes"
Oct 30 22:14:57.989962 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:14:57.989938    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cef9fb-a937-4586-ad66-b62771a6d36a" path="/var/lib/kubelet/pods/e9cef9fb-a937-4586-ad66-b62771a6d36a/volumes"
Oct 30 22:15:00.070341 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.070294    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"]
Oct 30 22:15:00.071402 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.071367    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr" podUID="d72dcd00-483a-42f3-90dd-14badea68753" containerName="maas-api" containerID="cri-o://63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca" gracePeriod=30
Oct 30 22:15:00.339788 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.339754    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 22:15:00.400205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.399550    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls\") pod \"d72dcd00-483a-42f3-90dd-14badea68753\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") "
Oct 30 22:15:00.400205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.399607    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d72dcd00-483a-42f3-90dd-14badea68753-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72dcd00-483a-42f3-90dd-14badea68753-openshift-service-ca\") pod \"d72dcd00-483a-42f3-90dd-14badea68753\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") "
Oct 30 22:15:00.400205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.399643    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72dcd00-483a-42f3-90dd-14badea68753-kube-api-access-nh4tv\" (UniqueName: \"kubernetes.io/projected/d72dcd00-483a-42f3-90dd-14badea68753-kube-api-access-nh4tv\") pod \"d72dcd00-483a-42f3-90dd-14badea68753\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") "
Oct 30 22:15:00.400205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.399682    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-metrics-certs\") pod \"d72dcd00-483a-42f3-90dd-14badea68753\" (UID: \"d72dcd00-483a-42f3-90dd-14badea68753\") "
Oct 30 22:15:00.401579 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.401546    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72dcd00-483a-42f3-90dd-14badea68753-openshift-service-ca" pod "d72dcd00-483a-42f3-90dd-14badea68753" (UID: "d72dcd00-483a-42f3-90dd-14badea68753"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:00.405738 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.405694    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72dcd00-483a-42f3-90dd-14badea68753-kube-api-access-nh4tv" pod "d72dcd00-483a-42f3-90dd-14badea68753" (UID: "d72dcd00-483a-42f3-90dd-14badea68753"). InnerVolumeSpecName "kube-api-access-nh4tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:00.429809 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.429772    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-metrics-certs" pod "d72dcd00-483a-42f3-90dd-14badea68753" (UID: "d72dcd00-483a-42f3-90dd-14badea68753"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:00.430031 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.429999    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls" pod "d72dcd00-483a-42f3-90dd-14badea68753" (UID: "d72dcd00-483a-42f3-90dd-14badea68753"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:00.500593 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.500556    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:00.500809 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.500609    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d72dcd00-483a-42f3-90dd-14badea68753-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:00.500809 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.500620    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh4tv\" (UniqueName: \"kubernetes.io/projected/d72dcd00-483a-42f3-90dd-14badea68753-kube-api-access-nh4tv\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:00.500809 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.500630    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d72dcd00-483a-42f3-90dd-14badea68753-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:00.717145 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.717104    5886 generic.go:356] "Generic (PLEG): container finished" podID="d72dcd00-483a-42f3-90dd-14badea68753" containerID="63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca" exitCode=0
Oct 30 22:15:00.717375 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.717197    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"
Oct 30 22:15:00.717375 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.717194    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr" event={"ID":"d72dcd00-483a-42f3-90dd-14badea68753","Type":"ContainerDied","Data":"63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca"}
Oct 30 22:15:00.717375 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.717282    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr" event={"ID":"d72dcd00-483a-42f3-90dd-14badea68753","Type":"ContainerDied","Data":"a14c1e3dfbbeb797af8530918f3323b6361c7bb93ee1a0fa98060cf57b9e3510"}
Oct 30 22:15:00.717375 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.717306    5886 scope.go:122] "RemoveContainer" containerID="63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca"
Oct 30 22:15:00.727786 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.727768    5886 scope.go:122] "RemoveContainer" containerID="63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca"
Oct 30 22:15:00.728074 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:00.728051    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca\": container with ID starting with 63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca not found: ID does not exist" containerID="63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca"
Oct 30 22:15:00.728126 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.728083    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca"} err="failed to get container status \"63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca\": rpc error: code = NotFound desc = could not find container \"63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca\": container with ID starting with 63ea7806624ccce8fd29f2324eac8bd2cce5e430c500bb716140217b522152ca not found: ID does not exist"
Oct 30 22:15:00.771454 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.771421    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"]
Oct 30 22:15:00.782726 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:00.782695    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b31518-8b8bdd8bb-98lwr"]
Oct 30 22:15:01.990081 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:01.990041    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72dcd00-483a-42f3-90dd-14badea68753" path="/var/lib/kubelet/pods/d72dcd00-483a-42f3-90dd-14badea68753/volumes"
Oct 30 22:15:07.836540 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:07.836502    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"]
Oct 30 22:15:07.837107 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:07.836678    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk" podUID="74b988a3-a45c-4eb6-8324-7ea24b1aa434" containerName="maas-api" containerID="cri-o://5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d" gracePeriod=30
Oct 30 22:15:08.089083 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.089012    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 22:15:08.184537 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.184497    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-metrics-certs\") pod \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") "
Oct 30 22:15:08.184740 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.184569    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b988a3-a45c-4eb6-8324-7ea24b1aa434-kube-api-access-lgxxm\" (UniqueName: \"kubernetes.io/projected/74b988a3-a45c-4eb6-8324-7ea24b1aa434-kube-api-access-lgxxm\") pod \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") "
Oct 30 22:15:08.184740 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.184599    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b988a3-a45c-4eb6-8324-7ea24b1aa434-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b988a3-a45c-4eb6-8324-7ea24b1aa434-openshift-service-ca\") pod \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") "
Oct 30 22:15:08.184740 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.184689    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-maas-api-tls\") pod \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\" (UID: \"74b988a3-a45c-4eb6-8324-7ea24b1aa434\") "
Oct 30 22:15:08.185118 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.185083    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b988a3-a45c-4eb6-8324-7ea24b1aa434-openshift-service-ca" pod "74b988a3-a45c-4eb6-8324-7ea24b1aa434" (UID: "74b988a3-a45c-4eb6-8324-7ea24b1aa434"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:08.186857 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.186823    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b988a3-a45c-4eb6-8324-7ea24b1aa434-kube-api-access-lgxxm" pod "74b988a3-a45c-4eb6-8324-7ea24b1aa434" (UID: "74b988a3-a45c-4eb6-8324-7ea24b1aa434"). InnerVolumeSpecName "kube-api-access-lgxxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:08.186971 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.186892    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-metrics-certs" pod "74b988a3-a45c-4eb6-8324-7ea24b1aa434" (UID: "74b988a3-a45c-4eb6-8324-7ea24b1aa434"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:08.187700 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.187683    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-maas-api-tls" pod "74b988a3-a45c-4eb6-8324-7ea24b1aa434" (UID: "74b988a3-a45c-4eb6-8324-7ea24b1aa434"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:08.285983 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.285946    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgxxm\" (UniqueName: \"kubernetes.io/projected/74b988a3-a45c-4eb6-8324-7ea24b1aa434-kube-api-access-lgxxm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:08.285983 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.285978    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74b988a3-a45c-4eb6-8324-7ea24b1aa434-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:08.285983 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.285988    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:08.285983 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.285996    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74b988a3-a45c-4eb6-8324-7ea24b1aa434-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:08.764643 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.764607    5886 generic.go:356] "Generic (PLEG): container finished" podID="74b988a3-a45c-4eb6-8324-7ea24b1aa434" containerID="5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d" exitCode=0
Oct 30 22:15:08.764817 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.764659    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"
Oct 30 22:15:08.764817 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.764692    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk" event={"ID":"74b988a3-a45c-4eb6-8324-7ea24b1aa434","Type":"ContainerDied","Data":"5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d"}
Oct 30 22:15:08.764817 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.764748    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk" event={"ID":"74b988a3-a45c-4eb6-8324-7ea24b1aa434","Type":"ContainerDied","Data":"93baf598ab60a8fa5ecb9080aba9484f63e5d921e0b1c564afe6bffb779f9e89"}
Oct 30 22:15:08.764817 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.764770    5886 scope.go:122] "RemoveContainer" containerID="5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d"
Oct 30 22:15:08.779769 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.779746    5886 scope.go:122] "RemoveContainer" containerID="5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d"
Oct 30 22:15:08.780150 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:08.780126    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d\": container with ID starting with 5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d not found: ID does not exist" containerID="5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d"
Oct 30 22:15:08.780240 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.780173    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d"} err="failed to get container status \"5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d\": rpc error: code = NotFound desc = could not find container \"5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d\": container with ID starting with 5981e6a1486e80b848589d25cacc74e4c0d36cf682162fcad4526efaf8fde69d not found: ID does not exist"
Oct 30 22:15:08.821581 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.821545    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"]
Oct 30 22:15:08.834293 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:08.834261    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64d142-bf648fff4-sdnzk"]
Oct 30 22:15:09.986924 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:09.986889    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b988a3-a45c-4eb6-8324-7ea24b1aa434" path="/var/lib/kubelet/pods/74b988a3-a45c-4eb6-8324-7ea24b1aa434/volumes"
Oct 30 22:15:11.033599 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:11.033561    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:15:11.516337 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:11.516295    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"]
Oct 30 22:15:16.037571 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.037532    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"]
Oct 30 22:15:16.038070 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.037788    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy" containerID="cri-o://6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324" gracePeriod=30
Oct 30 22:15:16.749100 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749061    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"]
Oct 30 22:15:16.749610 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749589    5886 cpu_manager.go:414] "containerMap: removing container" podUID="e625489f-694d-4ca8-95f6-3192e3e084e4" containerName="maas-api"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749611    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e625489f-694d-4ca8-95f6-3192e3e084e4" containerName="maas-api"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749632    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7ed45761-7d92-4890-9405-3743f30a43bb" containerName="maas-api"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749640    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed45761-7d92-4890-9405-3743f30a43bb" containerName="maas-api"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749652    5886 cpu_manager.go:414] "containerMap: removing container" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749660    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749672    5886 cpu_manager.go:414] "containerMap: removing container" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749679    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749694    5886 cpu_manager.go:414] "containerMap: removing container" podUID="98aee083-5553-48d7-a3d9-451c5588e455" containerName="payload-pre-processing"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749700    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98aee083-5553-48d7-a3d9-451c5588e455" containerName="payload-pre-processing"
Oct 30 22:15:16.749712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749712    5886 cpu_manager.go:414] "containerMap: removing container" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749718    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749727    5886 cpu_manager.go:414] "containerMap: removing container" podUID="5a39adb4-8217-40e0-9e5a-843e148a49c5" containerName="payload-pre-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749735    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a39adb4-8217-40e0-9e5a-843e148a49c5" containerName="payload-pre-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749746    5886 cpu_manager.go:414] "containerMap: removing container" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749753    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749764    5886 cpu_manager.go:414] "containerMap: removing container" podUID="1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749771    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749780    5886 cpu_manager.go:414] "containerMap: removing container" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749788    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749798    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749805    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749816    5886 cpu_manager.go:414] "containerMap: removing container" podUID="745272f1-fc0f-45ee-9e94-c5c9fc2583ed" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749823    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745272f1-fc0f-45ee-9e94-c5c9fc2583ed" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749834    5886 cpu_manager.go:414] "containerMap: removing container" podUID="d72dcd00-483a-42f3-90dd-14badea68753" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749841    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72dcd00-483a-42f3-90dd-14badea68753" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749850    5886 cpu_manager.go:414] "containerMap: removing container" podUID="2811cac3-c5ae-4ae5-988b-c4839e9c8364" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749857    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2811cac3-c5ae-4ae5-988b-c4839e9c8364" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749868    5886 cpu_manager.go:414] "containerMap: removing container" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749875    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749891    5886 cpu_manager.go:414] "containerMap: removing container" podUID="4ccd3396-e2ef-4338-aca4-3d3fe3296910" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749898    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccd3396-e2ef-4338-aca4-3d3fe3296910" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749909    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749915    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749925    5886 cpu_manager.go:414] "containerMap: removing container" podUID="e9cef9fb-a937-4586-ad66-b62771a6d36a" containerName="payload-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749932    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cef9fb-a937-4586-ad66-b62771a6d36a" containerName="payload-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749946    5886 cpu_manager.go:414] "containerMap: removing container" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749954    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749965    5886 cpu_manager.go:414] "containerMap: removing container" podUID="1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" containerName="payload-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749972    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" containerName="payload-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749985    5886 cpu_manager.go:414] "containerMap: removing container" podUID="78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.749992    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750009    5886 cpu_manager.go:414] "containerMap: removing container" podUID="7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750017    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750030    5886 cpu_manager.go:414] "containerMap: removing container" podUID="415f6ded-e8b1-4004-8c65-fa9aecaf195b" containerName="discovery"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750037    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415f6ded-e8b1-4004-8c65-fa9aecaf195b" containerName="discovery"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750063    5886 cpu_manager.go:414] "containerMap: removing container" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750071    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750086    5886 cpu_manager.go:414] "containerMap: removing container" podUID="53d2c15f-71ee-4335-bf9d-50632aab2c62" containerName="payload-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750093    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53d2c15f-71ee-4335-bf9d-50632aab2c62" containerName="payload-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750105    5886 cpu_manager.go:414] "containerMap: removing container" podUID="c2ceb265-b14e-4cbc-a8a4-9ea026496943" containerName="payload-pre-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750113    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ceb265-b14e-4cbc-a8a4-9ea026496943" containerName="payload-pre-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750122    5886 cpu_manager.go:414] "containerMap: removing container" podUID="01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750129    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750140    5886 cpu_manager.go:414] "containerMap: removing container" podUID="74b988a3-a45c-4eb6-8324-7ea24b1aa434" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750146    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b988a3-a45c-4eb6-8324-7ea24b1aa434" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750170    5886 cpu_manager.go:414] "containerMap: removing container" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750178    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750187    5886 cpu_manager.go:414] "containerMap: removing container" podUID="ff58ecb3-8871-46a9-aa79-a5067e38bb98" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750195    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff58ecb3-8871-46a9-aa79-a5067e38bb98" containerName="maas-api"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750205    5886 cpu_manager.go:414] "containerMap: removing container" podUID="21651e7e-fce1-4e60-b50a-20b0ce4f77b9" containerName="payload-processing"
Oct 30 22:15:16.750280 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750212    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21651e7e-fce1-4e60-b50a-20b0ce4f77b9" containerName="payload-processing"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750325    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf54c5f-a5f8-421a-be0e-c75ee4906734" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750343    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="98aee083-5553-48d7-a3d9-451c5588e455" containerName="payload-pre-processing"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750353    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab4d127-ceba-4fe7-9ff0-3f7a4d6e7b7a" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750364    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a39adb4-8217-40e0-9e5a-843e148a49c5" containerName="payload-pre-processing"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750374    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="970e4f4a-4008-4139-a720-0c6d8e1bf0b7" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750384    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3a4d7f-1c3e-4705-9b8a-bdfa32012f9e" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750392    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb6383b-3962-43f7-be89-29de959289ea" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750402    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f1c731-26c5-4ec4-abd8-458909a6ff6e" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750411    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="745272f1-fc0f-45ee-9e94-c5c9fc2583ed" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750419    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72dcd00-483a-42f3-90dd-14badea68753" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750427    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="2811cac3-c5ae-4ae5-988b-c4839e9c8364" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750434    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750443    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccd3396-e2ef-4338-aca4-3d3fe3296910" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750453    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1254d30-70fd-4247-b20b-18233b6a6992" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750461    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cef9fb-a937-4586-ad66-b62771a6d36a" containerName="payload-processing"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750471    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5962f18-d7e2-4dc4-83c9-a1f3d784b59e" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750480    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6fb98f-3e4b-4612-a02c-5f554ff5b7f2" containerName="payload-processing"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750488    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d2b79f-b40a-411d-bf1c-9d5a101b8e7a" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750498    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed962bd-2ed6-44b4-a4e3-bef92fc054b8" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750507    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="415f6ded-e8b1-4004-8c65-fa9aecaf195b" containerName="discovery"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750518    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750529    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b1ff52-2fef-4268-93b4-52325d4db4ff" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750539    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="53d2c15f-71ee-4335-bf9d-50632aab2c62" containerName="payload-processing"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750550    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ceb265-b14e-4cbc-a8a4-9ea026496943" containerName="payload-pre-processing"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750559    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750567    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ffc8a1-8d4a-41cf-be78-7c82a8e5ad2e" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750576    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b988a3-a45c-4eb6-8324-7ea24b1aa434" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750585    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e7bf60-4cf6-4ce8-bfc2-7f6cea4f559d" containerName="istio-proxy"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750593    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff58ecb3-8871-46a9-aa79-a5067e38bb98" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750609    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="21651e7e-fce1-4e60-b50a-20b0ce4f77b9" containerName="payload-processing"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750619    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="e625489f-694d-4ca8-95f6-3192e3e084e4" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750627    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed45761-7d92-4890-9405-3743f30a43bb" containerName="maas-api"
Oct 30 22:15:16.752549 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.750636    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy"
Oct 30 22:15:16.755559 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.755539    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:16.770441 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.770420    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:16.781709 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.781679    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zxbhb\""
Oct 30 22:15:16.785540 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.785501    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"]
Oct 30 22:15:16.880331 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.880295    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phn2s\" (UniqueName: \"kubernetes.io/projected/5491fd60-96f5-4887-8674-09491ff391c8-kube-api-access-phn2s\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:16.880331 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.880339    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:16.880572 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.880422    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:16.981291 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.981246    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phn2s\" (UniqueName: \"kubernetes.io/projected/5491fd60-96f5-4887-8674-09491ff391c8-kube-api-access-phn2s\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:16.981291 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.981298    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:16.981522 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.981335    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:16.982035 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.982008    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:16.983743 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:16.983718    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:17.004603 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:17.004538    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phn2s\" (UniqueName: \"kubernetes.io/projected/5491fd60-96f5-4887-8674-09491ff391c8-kube-api-access-phn2s\") pod \"maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:17.069530 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:17.069500    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:17.096311 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:17.096278    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"]
Oct 30 22:15:17.134347 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:17.134314    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"]
Oct 30 22:15:17.393417 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:17.393328    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"]
Oct 30 22:15:17.397385 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:15:17.397357    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5491fd60_96f5_4887_8674_09491ff391c8.slice/crio-f3d1a3ccd0305fe11694b1e7b825dffb83702cdd69bf288e68fcf84fe70b4f93 WatchSource:0}: Error finding container f3d1a3ccd0305fe11694b1e7b825dffb83702cdd69bf288e68fcf84fe70b4f93: Status 404 returned error can't find the container with id f3d1a3ccd0305fe11694b1e7b825dffb83702cdd69bf288e68fcf84fe70b4f93
Oct 30 22:15:17.822821 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:17.822784    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9" event={"ID":"5491fd60-96f5-4887-8674-09491ff391c8","Type":"ContainerStarted","Data":"9631f70e4d914624cf891366b7ae2d5a144f3c485dd5e01bfb1b1cdadb5deda7"}
Oct 30 22:15:17.823017 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:17.822840    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9" event={"ID":"5491fd60-96f5-4887-8674-09491ff391c8","Type":"ContainerStarted","Data":"f3d1a3ccd0305fe11694b1e7b825dffb83702cdd69bf288e68fcf84fe70b4f93"}
Oct 30 22:15:17.862342 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:17.862289    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9" podStartSLOduration=1.862275771 podStartE2EDuration="1.862275771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:17.861903388 +0000 UTC m=+3917.991154844" watchObservedRunningTime="2026-10-30 22:15:17.862275771 +0000 UTC m=+3917.991527227"
Oct 30 22:15:18.052539 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:18.052446    5886 patch_prober.go:29] interesting pod/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:18.052539 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:18.052505    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:18.830406 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:18.830366    5886 generic.go:356] "Generic (PLEG): container finished" podID="5491fd60-96f5-4887-8674-09491ff391c8" containerID="9631f70e4d914624cf891366b7ae2d5a144f3c485dd5e01bfb1b1cdadb5deda7" exitCode=0
Oct 30 22:15:18.830811 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:18.830456    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9" event={"ID":"5491fd60-96f5-4887-8674-09491ff391c8","Type":"ContainerDied","Data":"9631f70e4d914624cf891366b7ae2d5a144f3c485dd5e01bfb1b1cdadb5deda7"}
Oct 30 22:15:19.992279 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:19.992246    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:20.115491 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.115388    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5491fd60-96f5-4887-8674-09491ff391c8-kube-api-access-phn2s\" (UniqueName: \"kubernetes.io/projected/5491fd60-96f5-4887-8674-09491ff391c8-kube-api-access-phn2s\") pod \"5491fd60-96f5-4887-8674-09491ff391c8\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") "
Oct 30 22:15:20.115491 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.115461    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-openshift-ca\") pod \"5491fd60-96f5-4887-8674-09491ff391c8\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") "
Oct 30 22:15:20.115731 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.115584    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-xks-ca\") pod \"5491fd60-96f5-4887-8674-09491ff391c8\" (UID: \"5491fd60-96f5-4887-8674-09491ff391c8\") "
Oct 30 22:15:20.115870 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.115840    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-openshift-ca" pod "5491fd60-96f5-4887-8674-09491ff391c8" (UID: "5491fd60-96f5-4887-8674-09491ff391c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.116054 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.116021    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-openshift-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:20.117690 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.117668    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5491fd60-96f5-4887-8674-09491ff391c8-kube-api-access-phn2s" pod "5491fd60-96f5-4887-8674-09491ff391c8" (UID: "5491fd60-96f5-4887-8674-09491ff391c8"). InnerVolumeSpecName "kube-api-access-phn2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.117794 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.117706    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-xks-ca" pod "5491fd60-96f5-4887-8674-09491ff391c8" (UID: "5491fd60-96f5-4887-8674-09491ff391c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:20.217576 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.217540    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5491fd60-96f5-4887-8674-09491ff391c8-maas-api-xks-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:20.217576 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.217570    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phn2s\" (UniqueName: \"kubernetes.io/projected/5491fd60-96f5-4887-8674-09491ff391c8-kube-api-access-phn2s\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:20.844169 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.844115    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"
Oct 30 22:15:20.844169 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.844135    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9" event={"ID":"5491fd60-96f5-4887-8674-09491ff391c8","Type":"ContainerDied","Data":"f3d1a3ccd0305fe11694b1e7b825dffb83702cdd69bf288e68fcf84fe70b4f93"}
Oct 30 22:15:20.844417 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:20.844220    5886 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3d1a3ccd0305fe11694b1e7b825dffb83702cdd69bf288e68fcf84fe70b4f93"
Oct 30 22:15:21.290061 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.290032    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 22:15:21.428572 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428478    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-podinfo\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.428726 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428601    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-certs\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.428726 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428663    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istiod-ca-cert\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.428726 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428705    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-socket\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.428881 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428732    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-data\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.428881 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428782    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-credential-socket\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.428881 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428815    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-envoy\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.429049 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428881    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-token\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-token\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.429049 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428914    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-kube-api-access-rtk2t\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-kube-api-access-rtk2t\") pod \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\" (UID: \"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7\") "
Oct 30 22:15:21.429183 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.428907    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-certs" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:21.429242 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429047    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-credential-socket" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:21.429242 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429061    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-data" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:21.429242 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429074    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-socket" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:21.429242 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429189    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istiod-ca-cert" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.429456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429377    5886 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.429456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429399    5886 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istiod-ca-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.429456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429413    5886 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-workload-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.429456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429425    5886 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-data\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.429456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.429438    5886 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-credential-socket\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.431456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.431316    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-podinfo" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:21.431599 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.431563    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-token" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.431722 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.431699    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-envoy" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:21.431937 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.431918    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-kube-api-access-rtk2t" pod "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" (UID: "16257b19-c7b1-4dfc-bcef-e21e9d8c63f7"). InnerVolumeSpecName "kube-api-access-rtk2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.530814 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.530782    5886 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.530814 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.530811    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtk2t\" (UniqueName: \"kubernetes.io/projected/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-kube-api-access-rtk2t\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.530814 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.530824    5886 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-podinfo\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.531059 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.530836    5886 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7-istio-envoy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:21.571540 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.571495    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"]
Oct 30 22:15:21.852125 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.852033    5886 generic.go:356] "Generic (PLEG): container finished" podID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerID="6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324" exitCode=0
Oct 30 22:15:21.852125 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.852113    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"
Oct 30 22:15:21.852379 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.852113    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" event={"ID":"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7","Type":"ContainerDied","Data":"6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324"}
Oct 30 22:15:21.852379 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.852225    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr" event={"ID":"16257b19-c7b1-4dfc-bcef-e21e9d8c63f7","Type":"ContainerDied","Data":"de84bf8f694aa122b9b377c2fbb3c7d844780622b73c88cf507c67bbe26d8b94"}
Oct 30 22:15:21.852379 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.852258    5886 scope.go:122] "RemoveContainer" containerID="6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324"
Oct 30 22:15:21.863556 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.863537    5886 scope.go:122] "RemoveContainer" containerID="c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4"
Oct 30 22:15:21.872449 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.872414    5886 scope.go:122] "RemoveContainer" containerID="6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324"
Oct 30 22:15:21.872756 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:21.872736    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324\": container with ID starting with 6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324 not found: ID does not exist" containerID="6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324"
Oct 30 22:15:21.872829 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.872763    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324"} err="failed to get container status \"6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324\": rpc error: code = NotFound desc = could not find container \"6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324\": container with ID starting with 6ca0269401e7f06f0e12a8b8a7e8f5d32e18e8354617566b7b43cfa5b34f7324 not found: ID does not exist"
Oct 30 22:15:21.872829 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.872784    5886 scope.go:122] "RemoveContainer" containerID="c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4"
Oct 30 22:15:21.873071 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:21.873053    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4\": container with ID starting with c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4 not found: ID does not exist" containerID="c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4"
Oct 30 22:15:21.873129 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.873077    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4"} err="failed to get container status \"c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4\": rpc error: code = NotFound desc = could not find container \"c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4\": container with ID starting with c90aec43acd25de4e3abd13a4179f4f5c6e659073b160f27fb78c6d26cd08ef4 not found: ID does not exist"
Oct 30 22:15:21.907454 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.907425    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"]
Oct 30 22:15:21.918336 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.918303    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-64d142-openshift-default-66b64d8c87-z62vr"]
Oct 30 22:15:21.986978 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:21.986943    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" path="/var/lib/kubelet/pods/16257b19-c7b1-4dfc-bcef-e21e9d8c63f7/volumes"
Oct 30 22:15:22.097030 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:22.096965    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6" podUID="6ed04681-8f10-4915-85ba-d0df6be22c13" containerName="payload-processing" containerID="cri-o://c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8" gracePeriod=25
Oct 30 22:15:22.135760 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:22.135673    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2" podUID="5ee44288-11ec-48fd-aa6f-8f83e27b27a1" containerName="payload-pre-processing" containerID="cri-o://416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb" gracePeriod=25
Oct 30 22:15:26.536812 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.536769    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main" containerID="cri-o://b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764" gracePeriod=45
Oct 30 22:15:26.794313 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.794244    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 22:15:26.884528 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.884491    5886 generic.go:356] "Generic (PLEG): container finished" podID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerID="b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764" exitCode=0
Oct 30 22:15:26.884737 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.884563    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"
Oct 30 22:15:26.884737 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.884583    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" event={"ID":"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0","Type":"ContainerDied","Data":"b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764"}
Oct 30 22:15:26.884737 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.884632    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl" event={"ID":"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0","Type":"ContainerDied","Data":"184886ef842f3f0475f83fa084a924f2fc600b85774ba9c07bf920921547ed21"}
Oct 30 22:15:26.884737 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.884654    5886 scope.go:122] "RemoveContainer" containerID="b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764"
Oct 30 22:15:26.885236 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.885212    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-model-cache\") pod \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") "
Oct 30 22:15:26.885386 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.885336    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tmp-dir\") pod \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") "
Oct 30 22:15:26.885459 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.885405    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tls-certs\") pod \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") "
Oct 30 22:15:26.885459 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.885448    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-home\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-home\") pod \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") "
Oct 30 22:15:26.885570 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.885478    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-dshm\") pod \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") "
Oct 30 22:15:26.885570 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.885502    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-model-cache" pod "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" (UID: "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.885570 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.885511    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-kube-api-access-tlgtz\" (UniqueName: \"kubernetes.io/projected/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-kube-api-access-tlgtz\") pod \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\" (UID: \"b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0\") "
Oct 30 22:15:26.885956 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.885863    5886 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.886235 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.886194    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-home" pod "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" (UID: "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.886439 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.886416    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tmp-dir" pod "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" (UID: "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.888387 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.888361    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tls-certs" pod "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" (UID: "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:26.888613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.888587    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-kube-api-access-tlgtz" pod "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" (UID: "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0"). InnerVolumeSpecName "kube-api-access-tlgtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:26.890001 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.889976    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-dshm" pod "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" (UID: "b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:26.911483 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.911456    5886 scope.go:122] "RemoveContainer" containerID="b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764"
Oct 30 22:15:26.912010 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:26.911936    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764\": container with ID starting with b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764 not found: ID does not exist" containerID="b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764"
Oct 30 22:15:26.912010 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.911981    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764"} err="failed to get container status \"b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764\": rpc error: code = NotFound desc = could not find container \"b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764\": container with ID starting with b803c9ea12da41a9bbf910da19d1ff5a3303a3c336e26ffb8033d06baca5f764 not found: ID does not exist"
Oct 30 22:15:26.987091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.987055    5886 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.987091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.987088    5886 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.987091 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.987099    5886 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.987363 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.987111    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlgtz\" (UniqueName: \"kubernetes.io/projected/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-kube-api-access-tlgtz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:26.987363 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:26.987123    5886 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:27.238442 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:27.238403    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"]
Oct 30 22:15:27.250764 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:27.250731    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ea40ee/distinct-w4-ea40ee-kserve-5cfd44b8b9-66zbl"]
Oct 30 22:15:27.988814 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:27.988773    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" path="/var/lib/kubelet/pods/b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0/volumes"
Oct 30 22:15:28.099258 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.099219    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"]
Oct 30 22:15:28.100335 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.100296    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb" podUID="12524c2e-b65b-4b38-aea2-b0e7922e2d14" containerName="maas-api" containerID="cri-o://62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b" gracePeriod=30
Oct 30 22:15:28.386575 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.386546    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 22:15:28.504282 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.504250    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-maas-api-tls\") pod \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") "
Oct 30 22:15:28.504456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.504338    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12524c2e-b65b-4b38-aea2-b0e7922e2d14-kube-api-access-5dnpb\" (UniqueName: \"kubernetes.io/projected/12524c2e-b65b-4b38-aea2-b0e7922e2d14-kube-api-access-5dnpb\") pod \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") "
Oct 30 22:15:28.504456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.504362    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12524c2e-b65b-4b38-aea2-b0e7922e2d14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12524c2e-b65b-4b38-aea2-b0e7922e2d14-openshift-service-ca\") pod \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") "
Oct 30 22:15:28.504456 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.504449    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-metrics-certs\") pod \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\" (UID: \"12524c2e-b65b-4b38-aea2-b0e7922e2d14\") "
Oct 30 22:15:28.504879 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.504839    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12524c2e-b65b-4b38-aea2-b0e7922e2d14-openshift-service-ca" pod "12524c2e-b65b-4b38-aea2-b0e7922e2d14" (UID: "12524c2e-b65b-4b38-aea2-b0e7922e2d14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:28.506613 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.506572    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-metrics-certs" pod "12524c2e-b65b-4b38-aea2-b0e7922e2d14" (UID: "12524c2e-b65b-4b38-aea2-b0e7922e2d14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:28.506764 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.506727    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12524c2e-b65b-4b38-aea2-b0e7922e2d14-kube-api-access-5dnpb" pod "12524c2e-b65b-4b38-aea2-b0e7922e2d14" (UID: "12524c2e-b65b-4b38-aea2-b0e7922e2d14"). InnerVolumeSpecName "kube-api-access-5dnpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:28.506843 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.506801    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-maas-api-tls" pod "12524c2e-b65b-4b38-aea2-b0e7922e2d14" (UID: "12524c2e-b65b-4b38-aea2-b0e7922e2d14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:28.606629 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.606533    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dnpb\" (UniqueName: \"kubernetes.io/projected/12524c2e-b65b-4b38-aea2-b0e7922e2d14-kube-api-access-5dnpb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:28.606629 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.606567    5886 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12524c2e-b65b-4b38-aea2-b0e7922e2d14-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:28.606629 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.606581    5886 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-metrics-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:28.606629 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.606593    5886 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12524c2e-b65b-4b38-aea2-b0e7922e2d14-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:28.899482 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.899442    5886 generic.go:356] "Generic (PLEG): container finished" podID="12524c2e-b65b-4b38-aea2-b0e7922e2d14" containerID="62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b" exitCode=0
Oct 30 22:15:28.899653 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.899495    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"
Oct 30 22:15:28.899653 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.899524    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb" event={"ID":"12524c2e-b65b-4b38-aea2-b0e7922e2d14","Type":"ContainerDied","Data":"62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b"}
Oct 30 22:15:28.899653 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.899583    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb" event={"ID":"12524c2e-b65b-4b38-aea2-b0e7922e2d14","Type":"ContainerDied","Data":"124e4504be3750edbf5f73c6a9f63123c751e9d3897c5dcd8e6ba4eff10dfd82"}
Oct 30 22:15:28.899653 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.899609    5886 scope.go:122] "RemoveContainer" containerID="62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b"
Oct 30 22:15:28.909849 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.909830    5886 scope.go:122] "RemoveContainer" containerID="62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b"
Oct 30 22:15:28.910112 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:28.910094    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b\": container with ID starting with 62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b not found: ID does not exist" containerID="62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b"
Oct 30 22:15:28.910201 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.910122    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b"} err="failed to get container status \"62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b\": rpc error: code = NotFound desc = could not find container \"62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b\": container with ID starting with 62276e408e28feb3c11abb52c0c57b68fe85b3550ddfa1bad3ab623ee2032d2b not found: ID does not exist"
Oct 30 22:15:28.950822 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.950788    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"]
Oct 30 22:15:28.962238 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:28.962211    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ea40ee-86b9d75fc9-ldlxb"]
Oct 30 22:15:29.989591 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:29.989555    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12524c2e-b65b-4b38-aea2-b0e7922e2d14" path="/var/lib/kubelet/pods/12524c2e-b65b-4b38-aea2-b0e7922e2d14/volumes"
Oct 30 22:15:30.573111 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:30.573066    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6" podUID="6ed04681-8f10-4915-85ba-d0df6be22c13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:30.573111 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:30.573099    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2" podUID="5ee44288-11ec-48fd-aa6f-8f83e27b27a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:36.591329 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.591270    5886 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main" containerID="cri-o://b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8" gracePeriod=45
Oct 30 22:15:36.881949 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.881923    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 22:15:36.949571 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.949525    5886 generic.go:356] "Generic (PLEG): container finished" podID="45416ce3-5386-4cbe-a463-506a0e16c627" containerID="b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8" exitCode=0
Oct 30 22:15:36.949757 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.949574    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"
Oct 30 22:15:36.949757 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.949583    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" event={"ID":"45416ce3-5386-4cbe-a463-506a0e16c627","Type":"ContainerDied","Data":"b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8"}
Oct 30 22:15:36.949757 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.949622    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5" event={"ID":"45416ce3-5386-4cbe-a463-506a0e16c627","Type":"ContainerDied","Data":"9063fce699ac11dd9b1f6422a36dce2b03b6f637cacd3742792e8a32f90e089b"}
Oct 30 22:15:36.949757 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.949640    5886 scope.go:122] "RemoveContainer" containerID="b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8"
Oct 30 22:15:36.959769 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.959741    5886 scope.go:122] "RemoveContainer" containerID="b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8"
Oct 30 22:15:36.960119 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:36.960089    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8\": container with ID starting with b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8 not found: ID does not exist" containerID="b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8"
Oct 30 22:15:36.960284 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.960129    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8"} err="failed to get container status \"b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8\": rpc error: code = NotFound desc = could not find container \"b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8\": container with ID starting with b038d1b9a84f9b12b68ea4420480913cba826cc3f492bfe1594368f677775dd8 not found: ID does not exist"
Oct 30 22:15:36.992987 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.992947    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-home\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-home\") pod \"45416ce3-5386-4cbe-a463-506a0e16c627\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") "
Oct 30 22:15:36.993252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993043    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45416ce3-5386-4cbe-a463-506a0e16c627-kube-api-access-r57rx\" (UniqueName: \"kubernetes.io/projected/45416ce3-5386-4cbe-a463-506a0e16c627-kube-api-access-r57rx\") pod \"45416ce3-5386-4cbe-a463-506a0e16c627\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") "
Oct 30 22:15:36.993252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993102    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-model-cache\") pod \"45416ce3-5386-4cbe-a463-506a0e16c627\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") "
Oct 30 22:15:36.993252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993150    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45416ce3-5386-4cbe-a463-506a0e16c627-tls-certs\" (UniqueName: \"kubernetes.io/secret/45416ce3-5386-4cbe-a463-506a0e16c627-tls-certs\") pod \"45416ce3-5386-4cbe-a463-506a0e16c627\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") "
Oct 30 22:15:36.993252 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993241    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-dshm\") pod \"45416ce3-5386-4cbe-a463-506a0e16c627\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") "
Oct 30 22:15:36.993481 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993273    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-home" pod "45416ce3-5386-4cbe-a463-506a0e16c627" (UID: "45416ce3-5386-4cbe-a463-506a0e16c627"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.993481 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993286    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-tmp-dir\") pod \"45416ce3-5386-4cbe-a463-506a0e16c627\" (UID: \"45416ce3-5386-4cbe-a463-506a0e16c627\") "
Oct 30 22:15:36.993481 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993401    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-model-cache" pod "45416ce3-5386-4cbe-a463-506a0e16c627" (UID: "45416ce3-5386-4cbe-a463-506a0e16c627"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.993647 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993590    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-tmp-dir" pod "45416ce3-5386-4cbe-a463-506a0e16c627" (UID: "45416ce3-5386-4cbe-a463-506a0e16c627"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.993743 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993713    5886 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:36.993743 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993730    5886 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:36.993871 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.993742    5886 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:36.995714 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.995690    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45416ce3-5386-4cbe-a463-506a0e16c627-kube-api-access-r57rx" pod "45416ce3-5386-4cbe-a463-506a0e16c627" (UID: "45416ce3-5386-4cbe-a463-506a0e16c627"). InnerVolumeSpecName "kube-api-access-r57rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.995821 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.995726    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45416ce3-5386-4cbe-a463-506a0e16c627-tls-certs" pod "45416ce3-5386-4cbe-a463-506a0e16c627" (UID: "45416ce3-5386-4cbe-a463-506a0e16c627"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:36.995821 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:36.995747    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-dshm" pod "45416ce3-5386-4cbe-a463-506a0e16c627" (UID: "45416ce3-5386-4cbe-a463-506a0e16c627"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:37.094926 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:37.094892    5886 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45416ce3-5386-4cbe-a463-506a0e16c627-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.094926 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:37.094924    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r57rx\" (UniqueName: \"kubernetes.io/projected/45416ce3-5386-4cbe-a463-506a0e16c627-kube-api-access-r57rx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.095137 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:37.094935    5886 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45416ce3-5386-4cbe-a463-506a0e16c627-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:37.307545 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:37.307509    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"]
Oct 30 22:15:37.312494 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:37.312351    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9d677d/distinct-2-w2-9d677d-kserve-55b5fc6677-fnhz5"]
Oct 30 22:15:37.987359 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:37.987322    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" path="/var/lib/kubelet/pods/45416ce3-5386-4cbe-a463-506a0e16c627/volumes"
Oct 30 22:15:40.573289 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:40.573251    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6" podUID="6ed04681-8f10-4915-85ba-d0df6be22c13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:40.573717 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:40.573245    5886 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2" podUID="5ee44288-11ec-48fd-aa6f-8f83e27b27a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:15:40.838023 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:40.837920    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"]
Oct 30 22:15:40.849095 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:40.849060    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9d677d-r54c9"]
Oct 30 22:15:41.028167 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:41.028112    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:15:41.987478 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:41.987434    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5491fd60-96f5-4887-8674-09491ff391c8" path="/var/lib/kubelet/pods/5491fd60-96f5-4887-8674-09491ff391c8/volumes"
Oct 30 22:15:42.395849 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.395820    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 22:15:42.422908 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.422883    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 22:15:42.449855 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.449827    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed04681-8f10-4915-85ba-d0df6be22c13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed04681-8f10-4915-85ba-d0df6be22c13-plugins-config-volume\") pod \"6ed04681-8f10-4915-85ba-d0df6be22c13\" (UID: \"6ed04681-8f10-4915-85ba-d0df6be22c13\") "
Oct 30 22:15:42.449967 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.449860    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed04681-8f10-4915-85ba-d0df6be22c13-kube-api-access-rznxj\" (UniqueName: \"kubernetes.io/projected/6ed04681-8f10-4915-85ba-d0df6be22c13-kube-api-access-rznxj\") pod \"6ed04681-8f10-4915-85ba-d0df6be22c13\" (UID: \"6ed04681-8f10-4915-85ba-d0df6be22c13\") "
Oct 30 22:15:42.450204 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.450179    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed04681-8f10-4915-85ba-d0df6be22c13-plugins-config-volume" pod "6ed04681-8f10-4915-85ba-d0df6be22c13" (UID: "6ed04681-8f10-4915-85ba-d0df6be22c13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:42.451946 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.451926    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed04681-8f10-4915-85ba-d0df6be22c13-kube-api-access-rznxj" pod "6ed04681-8f10-4915-85ba-d0df6be22c13" (UID: "6ed04681-8f10-4915-85ba-d0df6be22c13"). InnerVolumeSpecName "kube-api-access-rznxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:42.550712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.550624    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-plugins-config-volume\") pod \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\" (UID: \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\") "
Oct 30 22:15:42.550712 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.550677    5886 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-kube-api-access-458xf\" (UniqueName: \"kubernetes.io/projected/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-kube-api-access-458xf\") pod \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\" (UID: \"5ee44288-11ec-48fd-aa6f-8f83e27b27a1\") "
Oct 30 22:15:42.550952 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.550932    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed04681-8f10-4915-85ba-d0df6be22c13-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.550996 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.550958    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rznxj\" (UniqueName: \"kubernetes.io/projected/6ed04681-8f10-4915-85ba-d0df6be22c13-kube-api-access-rznxj\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.551050 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.551030    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-plugins-config-volume" pod "5ee44288-11ec-48fd-aa6f-8f83e27b27a1" (UID: "5ee44288-11ec-48fd-aa6f-8f83e27b27a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:42.552985 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.552960    5886 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-kube-api-access-458xf" pod "5ee44288-11ec-48fd-aa6f-8f83e27b27a1" (UID: "5ee44288-11ec-48fd-aa6f-8f83e27b27a1"). InnerVolumeSpecName "kube-api-access-458xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:42.651809 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.651769    5886 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458xf\" (UniqueName: \"kubernetes.io/projected/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-kube-api-access-458xf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.651809 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.651797    5886 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ee44288-11ec-48fd-aa6f-8f83e27b27a1-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.990415 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.990378    5886 generic.go:356] "Generic (PLEG): container finished" podID="6ed04681-8f10-4915-85ba-d0df6be22c13" containerID="c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8" exitCode=0
Oct 30 22:15:42.990939 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.990419    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"
Oct 30 22:15:42.990939 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.990462    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6" event={"ID":"6ed04681-8f10-4915-85ba-d0df6be22c13","Type":"ContainerDied","Data":"c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8"}
Oct 30 22:15:42.990939 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.990519    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6" event={"ID":"6ed04681-8f10-4915-85ba-d0df6be22c13","Type":"ContainerDied","Data":"4cb1a2b14cd88a6837433b6d7a9e504ec98e457c976607d15e23085942db8ac4"}
Oct 30 22:15:42.990939 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.990538    5886 scope.go:122] "RemoveContainer" containerID="c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8"
Oct 30 22:15:42.992050 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.992024    5886 generic.go:356] "Generic (PLEG): container finished" podID="5ee44288-11ec-48fd-aa6f-8f83e27b27a1" containerID="416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb" exitCode=0
Oct 30 22:15:42.992201 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.992077    5886 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"
Oct 30 22:15:42.992201 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.992141    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2" event={"ID":"5ee44288-11ec-48fd-aa6f-8f83e27b27a1","Type":"ContainerDied","Data":"416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb"}
Oct 30 22:15:42.992326 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:42.992284    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2" event={"ID":"5ee44288-11ec-48fd-aa6f-8f83e27b27a1","Type":"ContainerDied","Data":"969f6f70361ed04f560ba9d2ad8ba7015caedf3eacb710df1c2ae369aa9dee05"}
Oct 30 22:15:43.001064 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.001047    5886 scope.go:122] "RemoveContainer" containerID="c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8"
Oct 30 22:15:43.001413 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:43.001392    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8\": container with ID starting with c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8 not found: ID does not exist" containerID="c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8"
Oct 30 22:15:43.001492 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.001420    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8"} err="failed to get container status \"c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8\": rpc error: code = NotFound desc = could not find container \"c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8\": container with ID starting with c6db71046a6d387bdd814cd48c0d6a47cc95fabcabca1c6dcb66af847eddc4b8 not found: ID does not exist"
Oct 30 22:15:43.001492 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.001454    5886 scope.go:122] "RemoveContainer" containerID="416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb"
Oct 30 22:15:43.011382 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.011361    5886 scope.go:122] "RemoveContainer" containerID="416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb"
Oct 30 22:15:43.011719 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:15:43.011696    5886 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb\": container with ID starting with 416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb not found: ID does not exist" containerID="416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb"
Oct 30 22:15:43.011774 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.011728    5886 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb"} err="failed to get container status \"416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb\": rpc error: code = NotFound desc = could not find container \"416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb\": container with ID starting with 416226cabfc22df10dcd84634fa8a567c248239315a445a00b1fee3e3fe86feb not found: ID does not exist"
Oct 30 22:15:43.040423 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.040384    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"]
Oct 30 22:15:43.052741 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.052694    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ea40ee-78c497bc79-qblt2"]
Oct 30 22:15:43.083903 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.083858    5886 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"]
Oct 30 22:15:43.088213 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.088171    5886 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ea40ee-5b86f47cc9-w9gl6"]
Oct 30 22:15:43.988124 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.988089    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee44288-11ec-48fd-aa6f-8f83e27b27a1" path="/var/lib/kubelet/pods/5ee44288-11ec-48fd-aa6f-8f83e27b27a1/volumes"
Oct 30 22:15:43.988527 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:43.988510    5886 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed04681-8f10-4915-85ba-d0df6be22c13" path="/var/lib/kubelet/pods/6ed04681-8f10-4915-85ba-d0df6be22c13/volumes"
Oct 30 22:15:49.616986 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:49.616955    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cc7x7_ab559120-5838-4b2d-9a35-c9c9d62c0dd7/manager/0.log"
Oct 30 22:15:49.806401 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:49.806363    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9snpv_e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83/kuadrant-console-plugin/0.log"
Oct 30 22:15:50.191838 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:50.191798    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-q6j5d_914c9e20-a98b-4655-8405-7dfbfc030564/manager/0.log"
Oct 30 22:15:50.691365 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:50.691329    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q7jpk_6f413ea5-f172-43ed-bb8f-425675362ac7/manager/0.log"
Oct 30 22:15:51.512876 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:51.512831    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-845f9f9b5f-ggr56_0ee59b1b-5575-4096-98e1-1f78882e72dc/manager/0.log"
Oct 30 22:15:51.728673 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:51.728639    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fmm47_584aa75a-57b4-40a0-98c9-dee63e23b032/manager/0.log"
Oct 30 22:15:52.899285 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:52.899227    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qw5z9_2d5eaaf1-15a5-4663-a680-4eb2107b1f75/server/0.log"
Oct 30 22:15:53.111804 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:15:53.111773    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ltdxn_12837eab-8ef2-4a9a-8423-fd13972fb1ad/manager/0.log"
Oct 30 22:16:04.627221 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:04.627185    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-845f9f9b5f-ggr56_0ee59b1b-5575-4096-98e1-1f78882e72dc/manager/0.log"
Oct 30 22:16:04.860273 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:04.860235    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fmm47_584aa75a-57b4-40a0-98c9-dee63e23b032/manager/0.log"
Oct 30 22:16:05.935424 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:05.935394    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qw5z9_2d5eaaf1-15a5-4663-a680-4eb2107b1f75/server/0.log"
Oct 30 22:16:06.125828 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:06.125794    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ltdxn_12837eab-8ef2-4a9a-8423-fd13972fb1ad/manager/0.log"
Oct 30 22:16:07.095671 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:07.095639    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b54f57f95-gs564_729b7547-83d6-4451-bde2-c54e7d019008/maas-api/0.log"
Oct 30 22:16:11.028258 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:16:11.028150    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:16:11.206439 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:11.206396    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cc7x7_ab559120-5838-4b2d-9a35-c9c9d62c0dd7/manager/0.log"
Oct 30 22:16:11.411567 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:11.411542    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9snpv_e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83/kuadrant-console-plugin/0.log"
Oct 30 22:16:11.763618 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:11.763513    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-q6j5d_914c9e20-a98b-4655-8405-7dfbfc030564/manager/0.log"
Oct 30 22:16:12.179546 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:12.179507    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q7jpk_6f413ea5-f172-43ed-bb8f-425675362ac7/manager/0.log"
Oct 30 22:16:13.365695 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:13.365662    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fccbd9b79-fzlvq_71550ced-41dd-4cab-adf5-ac8cee060005/kube-auth-proxy/0.log"
Oct 30 22:16:13.775306 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:13.775274    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-cf7lb_f9457827-beb1-4755-bb94-29ac9c2c1659/istio-proxy/1.log"
Oct 30 22:16:14.802944 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:14.802913    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9x9cn_9120e132-28e6-49ac-a31e-d263a71cea20/payload-pre-processing/0.log"
Oct 30 22:16:14.998624 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:14.998583    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-6l9b4_ee881cfc-ad3c-4017-8504-722b0543c67f/payload-processing/0.log"
Oct 30 22:16:15.647056 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:15.647022    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-cf7lb_f9457827-beb1-4755-bb94-29ac9c2c1659/istio-proxy/0.log"
Oct 30 22:16:16.200062 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:16.200027    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qbjmx_6b2f4c05-3262-4b06-adb2-d1ba47b8e1d3/main/0.log"
Oct 30 22:16:16.392802 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:16.392775    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-gp7xk_c79460cf-0d6b-41f0-9080-22bf5bd7a312/main/0.log"
Oct 30 22:16:16.584822 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:16.584738    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-d44dq_af485e41-5b2c-4b83-9bce-972e34da300d/main/0.log"
Oct 30 22:16:17.168592 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:17.168563    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-79kvw_0cebd872-e218-4bd4-a990-0c22fc0afb6e/main/0.log"
Oct 30 22:16:17.571846 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:17.571771    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-tvx5c_24915d9d-898e-4455-8c32-a4ff2bac9251/main/0.log"
Oct 30 22:16:28.424914 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.424870    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fx2nl/must-gather-x77fm"]
Oct 30 22:16:28.425532 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425510    5886 cpu_manager.go:414] "containerMap: removing container" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy"
Oct 30 22:16:28.425632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425535    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4631b858-9891-4eda-8a4a-8142682819b3" containerName="istio-proxy"
Oct 30 22:16:28.425632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425558    5886 cpu_manager.go:414] "containerMap: removing container" podUID="5491fd60-96f5-4887-8674-09491ff391c8" containerName="revoke-keys"
Oct 30 22:16:28.425632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425565    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5491fd60-96f5-4887-8674-09491ff391c8" containerName="revoke-keys"
Oct 30 22:16:28.425632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425582    5886 cpu_manager.go:414] "containerMap: removing container" podUID="5ee44288-11ec-48fd-aa6f-8f83e27b27a1" containerName="payload-pre-processing"
Oct 30 22:16:28.425632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425591    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee44288-11ec-48fd-aa6f-8f83e27b27a1" containerName="payload-pre-processing"
Oct 30 22:16:28.425632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425611    5886 cpu_manager.go:414] "containerMap: removing container" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy"
Oct 30 22:16:28.425632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425619    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy"
Oct 30 22:16:28.425632 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425629    5886 cpu_manager.go:414] "containerMap: removing container" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425636    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425655    5886 cpu_manager.go:414] "containerMap: removing container" podUID="6ed04681-8f10-4915-85ba-d0df6be22c13" containerName="payload-processing"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425663    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed04681-8f10-4915-85ba-d0df6be22c13" containerName="payload-processing"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425678    5886 cpu_manager.go:414] "containerMap: removing container" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425684    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425705    5886 cpu_manager.go:414] "containerMap: removing container" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425713    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d282f4-907f-4151-af9b-d5243ddc00ea" containerName="istio-proxy"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425730    5886 cpu_manager.go:414] "containerMap: removing container" podUID="12524c2e-b65b-4b38-aea2-b0e7922e2d14" containerName="maas-api"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425737    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12524c2e-b65b-4b38-aea2-b0e7922e2d14" containerName="maas-api"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425829    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="12524c2e-b65b-4b38-aea2-b0e7922e2d14" containerName="maas-api"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425841    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425851    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="5491fd60-96f5-4887-8674-09491ff391c8" containerName="revoke-keys"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425860    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee44288-11ec-48fd-aa6f-8f83e27b27a1" containerName="payload-pre-processing"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425871    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425880    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="45416ce3-5386-4cbe-a463-506a0e16c627" containerName="main"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425890    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed04681-8f10-4915-85ba-d0df6be22c13" containerName="payload-processing"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.425900    5886 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c42ccf-fbaa-4ec6-932c-1cc63c44b0b0" containerName="main"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.426018    5886 cpu_manager.go:414] "containerMap: removing container" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy"
Oct 30 22:16:28.426065 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.426027    5886 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16257b19-c7b1-4dfc-bcef-e21e9d8c63f7" containerName="istio-proxy"
Oct 30 22:16:28.429641 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.429619    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fx2nl/must-gather-x77fm"
Oct 30 22:16:28.443631 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.443610    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fx2nl\"/\"kube-root-ca.crt\""
Oct 30 22:16:28.446505 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.446487    5886 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fx2nl\"/\"openshift-service-ca.crt\""
Oct 30 22:16:28.457863 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.457838    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fx2nl/must-gather-x77fm"]
Oct 30 22:16:28.460925 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.460906    5886 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fx2nl\"/\"default-dockercfg-sh698\""
Oct 30 22:16:28.468384 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.468357    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/407f1491-35f9-4e4f-9a0e-4920c6370fbd-must-gather-output\") pod \"must-gather-x77fm\" (UID: \"407f1491-35f9-4e4f-9a0e-4920c6370fbd\") " pod="openshift-must-gather-fx2nl/must-gather-x77fm"
Oct 30 22:16:28.468517 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.468432    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbftb\" (UniqueName: \"kubernetes.io/projected/407f1491-35f9-4e4f-9a0e-4920c6370fbd-kube-api-access-rbftb\") pod \"must-gather-x77fm\" (UID: \"407f1491-35f9-4e4f-9a0e-4920c6370fbd\") " pod="openshift-must-gather-fx2nl/must-gather-x77fm"
Oct 30 22:16:28.569004 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.568972    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbftb\" (UniqueName: \"kubernetes.io/projected/407f1491-35f9-4e4f-9a0e-4920c6370fbd-kube-api-access-rbftb\") pod \"must-gather-x77fm\" (UID: \"407f1491-35f9-4e4f-9a0e-4920c6370fbd\") " pod="openshift-must-gather-fx2nl/must-gather-x77fm"
Oct 30 22:16:28.569204 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.569041    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/407f1491-35f9-4e4f-9a0e-4920c6370fbd-must-gather-output\") pod \"must-gather-x77fm\" (UID: \"407f1491-35f9-4e4f-9a0e-4920c6370fbd\") " pod="openshift-must-gather-fx2nl/must-gather-x77fm"
Oct 30 22:16:28.569378 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.569358    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/407f1491-35f9-4e4f-9a0e-4920c6370fbd-must-gather-output\") pod \"must-gather-x77fm\" (UID: \"407f1491-35f9-4e4f-9a0e-4920c6370fbd\") " pod="openshift-must-gather-fx2nl/must-gather-x77fm"
Oct 30 22:16:28.595354 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.595331    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbftb\" (UniqueName: \"kubernetes.io/projected/407f1491-35f9-4e4f-9a0e-4920c6370fbd-kube-api-access-rbftb\") pod \"must-gather-x77fm\" (UID: \"407f1491-35f9-4e4f-9a0e-4920c6370fbd\") " pod="openshift-must-gather-fx2nl/must-gather-x77fm"
Oct 30 22:16:28.743987 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:28.743907    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fx2nl/must-gather-x77fm"
Oct 30 22:16:29.043817 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:29.043724    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fx2nl/must-gather-x77fm"]
Oct 30 22:16:29.048328 ip-10-0-129-84 kubenswrapper[5886]: W1030 22:16:29.048299    5886 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod407f1491_35f9_4e4f_9a0e_4920c6370fbd.slice/crio-5cdb437ce8d21e0b5455957e6644c693bc79a73613803edf6785f933e6fe2a8f WatchSource:0}: Error finding container 5cdb437ce8d21e0b5455957e6644c693bc79a73613803edf6785f933e6fe2a8f: Status 404 returned error can't find the container with id 5cdb437ce8d21e0b5455957e6644c693bc79a73613803edf6785f933e6fe2a8f
Oct 30 22:16:29.053363 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:29.053341    5886 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:16:29.238232 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:29.238194    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/must-gather-x77fm" event={"ID":"407f1491-35f9-4e4f-9a0e-4920c6370fbd","Type":"ContainerStarted","Data":"5cdb437ce8d21e0b5455957e6644c693bc79a73613803edf6785f933e6fe2a8f"}
Oct 30 22:16:30.246592 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:30.246497    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/must-gather-x77fm" event={"ID":"407f1491-35f9-4e4f-9a0e-4920c6370fbd","Type":"ContainerStarted","Data":"9d606cbfa470166c7e9363a57db9c33859525c908cf13d8f7a2b3402db45043b"}
Oct 30 22:16:30.246592 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:30.246545    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/must-gather-x77fm" event={"ID":"407f1491-35f9-4e4f-9a0e-4920c6370fbd","Type":"ContainerStarted","Data":"face73e286388e7621a125735d1d0fd666af5c676f1d9005593f0080241354b9"}
Oct 30 22:16:30.283719 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:30.283662    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fx2nl/must-gather-x77fm" podStartSLOduration=1.4015326670000001 podStartE2EDuration="2.283644549s" totalImagesPullingTime="882.111882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:30.281917687 +0000 UTC m=+3990.411169145" watchObservedRunningTime="2026-10-30 22:16:30.283644549 +0000 UTC m=+3990.412896006"
Oct 30 22:16:33.064679 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:33.064636    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kmwrj_7e7da34c-23be-4c24-977d-8ef4b007fe8f/global-pull-secret-syncer/0.log"
Oct 30 22:16:33.784868 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:33.784817    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n9vj8_a3c22004-d04f-4ac5-ae97-224bf22bcf0e/konnectivity-agent/0.log"
Oct 30 22:16:34.027769 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:34.020352    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-84.ec2.internal_bd04ddc0cbac70d5dc4161808c80da69/haproxy/0.log"
Oct 30 22:16:35.361372 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:35.361292    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-845f9f9b5f-ggr56_0ee59b1b-5575-4096-98e1-1f78882e72dc/manager/0.log"
Oct 30 22:16:35.426108 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:35.425798    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fmm47_584aa75a-57b4-40a0-98c9-dee63e23b032/manager/0.log"
Oct 30 22:16:35.466412 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:35.462886    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-845f9f9b5f-ggr56_0ee59b1b-5575-4096-98e1-1f78882e72dc/manager/0.log"
Oct 30 22:16:35.507939 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:35.507906    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fmm47_584aa75a-57b4-40a0-98c9-dee63e23b032/manager/0.log"
Oct 30 22:16:37.450749 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:37.450715    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qw5z9_2d5eaaf1-15a5-4663-a680-4eb2107b1f75/server/0.log"
Oct 30 22:16:37.481910 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:37.481871    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qw5z9_2d5eaaf1-15a5-4663-a680-4eb2107b1f75/server/0.log"
Oct 30 22:16:39.956446 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:39.956402    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ltdxn_12837eab-8ef2-4a9a-8423-fd13972fb1ad/manager/0.log"
Oct 30 22:16:39.977133 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:39.977098    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ltdxn_12837eab-8ef2-4a9a-8423-fd13972fb1ad/manager/0.log"
Oct 30 22:16:40.665283 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:40.665248    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cc7x7_ab559120-5838-4b2d-9a35-c9c9d62c0dd7/manager/0.log"
Oct 30 22:16:40.786983 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:40.786955    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9snpv_e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83/kuadrant-console-plugin/0.log"
Oct 30 22:16:41.052399 ip-10-0-129-84 kubenswrapper[5886]: E1030 22:16:41.052297    5886 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gvjst" containerName="extractor"
Oct 30 22:16:41.067120 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:41.067084    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-q6j5d_914c9e20-a98b-4655-8405-7dfbfc030564/manager/0.log"
Oct 30 22:16:41.285178 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:41.282613    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-845f9f9b5f-ggr56_0ee59b1b-5575-4096-98e1-1f78882e72dc/manager/0.log"
Oct 30 22:16:41.326007 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:41.325933    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-845f9f9b5f-ggr56_0ee59b1b-5575-4096-98e1-1f78882e72dc/manager/0.log"
Oct 30 22:16:41.357231 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:41.357140    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fmm47_584aa75a-57b4-40a0-98c9-dee63e23b032/manager/0.log"
Oct 30 22:16:41.430977 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:41.430947    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fmm47_584aa75a-57b4-40a0-98c9-dee63e23b032/manager/0.log"
Oct 30 22:16:42.333652 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:42.333611    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q7jpk_6f413ea5-f172-43ed-bb8f-425675362ac7/manager/0.log"
Oct 30 22:16:43.201486 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.201455    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qw5z9_2d5eaaf1-15a5-4663-a680-4eb2107b1f75/server/0.log"
Oct 30 22:16:43.256427 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.256394    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qw5z9_2d5eaaf1-15a5-4663-a680-4eb2107b1f75/server/0.log"
Oct 30 22:16:43.626721 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.626626    5886 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"]
Oct 30 22:16:43.641221 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.640470    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.651781 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.651744    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"]
Oct 30 22:16:43.728146 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.728106    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-lib-modules\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.728424 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.728229    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-sys\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.728424 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.728350    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-proc\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.728424 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.728380    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-podres\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.728614 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.728479    5886 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mhv\" (UniqueName: \"kubernetes.io/projected/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-kube-api-access-s5mhv\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.829650 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.829606    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-proc\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.829650 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.829650    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-podres\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.829924 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.829706    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mhv\" (UniqueName: \"kubernetes.io/projected/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-kube-api-access-s5mhv\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.829924 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.829733    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-lib-modules\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.829924 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.829818    5886 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-sys\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.830101 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.829924    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-sys\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.830101 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.829988    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-proc\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.830101 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.830047    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-podres\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.830421 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.830398    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-lib-modules\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.851372 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.851314    5886 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mhv\" (UniqueName: \"kubernetes.io/projected/ece8cf70-bc7f-4ae8-9fc3-6095145ba236-kube-api-access-s5mhv\") pod \"perf-node-gather-daemonset-5kmk4\" (UID: \"ece8cf70-bc7f-4ae8-9fc3-6095145ba236\") " pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:43.986422 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:43.986382    5886 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:44.333893 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:44.333798    5886 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"]
Oct 30 22:16:44.387251 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:44.387195    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4" event={"ID":"ece8cf70-bc7f-4ae8-9fc3-6095145ba236","Type":"ContainerStarted","Data":"ecc7923842b5bd0c91170f2a239bb5097b85407cf03305bd46e4b1a42b195083"}
Oct 30 22:16:45.299268 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:45.299229    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ltdxn_12837eab-8ef2-4a9a-8423-fd13972fb1ad/manager/0.log"
Oct 30 22:16:45.393532 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:45.393494    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ltdxn_12837eab-8ef2-4a9a-8423-fd13972fb1ad/manager/0.log"
Oct 30 22:16:45.396207 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:45.396135    5886 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4" event={"ID":"ece8cf70-bc7f-4ae8-9fc3-6095145ba236","Type":"ContainerStarted","Data":"5345db6067ef5f48af9cd580e87782ea28e64bd0c7e839fbbaa92dc6fc0e6d4f"}
Oct 30 22:16:45.396413 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:45.396396    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:45.433648 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:45.433595    5886 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4" podStartSLOduration=2.433582697 podStartE2EDuration="2.433582697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:45.43269216 +0000 UTC m=+4005.561943617" watchObservedRunningTime="2026-10-30 22:16:45.433582697 +0000 UTC m=+4005.562834153"
Oct 30 22:16:45.867828 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:45.867790    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bnkf_d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/node-exporter/0.log"
Oct 30 22:16:45.987900 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:45.987839    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bnkf_d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/kube-rbac-proxy/0.log"
Oct 30 22:16:46.137320 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:46.137241    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bnkf_d4e1ba50-dfdf-4627-8ab6-97d9b1598d51/init-textfile/0.log"
Oct 30 22:16:46.838479 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:46.838449    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-845f9f9b5f-ggr56_0ee59b1b-5575-4096-98e1-1f78882e72dc/manager/0.log"
Oct 30 22:16:46.880088 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:46.880055    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fmm47_584aa75a-57b4-40a0-98c9-dee63e23b032/manager/0.log"
Oct 30 22:16:46.896419 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:46.896392    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-845f9f9b5f-ggr56_0ee59b1b-5575-4096-98e1-1f78882e72dc/manager/0.log"
Oct 30 22:16:46.940074 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:46.940046    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-fmm47_584aa75a-57b4-40a0-98c9-dee63e23b032/manager/0.log"
Oct 30 22:16:47.348450 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:47.348418    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e08318c-d1fa-4fcd-acd9-d9916c776598/prometheus/0.log"
Oct 30 22:16:47.477098 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:47.477066    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e08318c-d1fa-4fcd-acd9-d9916c776598/config-reloader/0.log"
Oct 30 22:16:47.595463 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:47.595435    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e08318c-d1fa-4fcd-acd9-d9916c776598/thanos-sidecar/0.log"
Oct 30 22:16:47.714767 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:47.714739    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e08318c-d1fa-4fcd-acd9-d9916c776598/kube-rbac-proxy-web/0.log"
Oct 30 22:16:47.832856 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:47.832822    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e08318c-d1fa-4fcd-acd9-d9916c776598/kube-rbac-proxy/0.log"
Oct 30 22:16:47.951115 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:47.951084    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e08318c-d1fa-4fcd-acd9-d9916c776598/kube-rbac-proxy-thanos/0.log"
Oct 30 22:16:48.095239 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:48.093102    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e08318c-d1fa-4fcd-acd9-d9916c776598/init-config-reloader/0.log"
Oct 30 22:16:48.223737 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:48.223705    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kfdst_66b15cb4-0694-40c5-aeac-939d1635a93f/prometheus-operator/0.log"
Oct 30 22:16:48.335779 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:48.335744    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kfdst_66b15cb4-0694-40c5-aeac-939d1635a93f/kube-rbac-proxy/0.log"
Oct 30 22:16:48.579302 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:48.579269    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59b7858fc8-fr72g_adf69dd8-b830-4e07-85d0-4950dcd2c9cf/telemeter-client/0.log"
Oct 30 22:16:48.699584 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:48.699534    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59b7858fc8-fr72g_adf69dd8-b830-4e07-85d0-4950dcd2c9cf/reload/0.log"
Oct 30 22:16:48.765642 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:48.765612    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qw5z9_2d5eaaf1-15a5-4663-a680-4eb2107b1f75/server/0.log"
Oct 30 22:16:48.817288 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:48.817218    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59b7858fc8-fr72g_adf69dd8-b830-4e07-85d0-4950dcd2c9cf/kube-rbac-proxy/0.log"
Oct 30 22:16:49.256017 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:49.255984    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-qw5z9_2d5eaaf1-15a5-4663-a680-4eb2107b1f75/server/0.log"
Oct 30 22:16:50.261830 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:50.261790    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ltdxn_12837eab-8ef2-4a9a-8423-fd13972fb1ad/manager/0.log"
Oct 30 22:16:50.734975 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:50.734933    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ltdxn_12837eab-8ef2-4a9a-8423-fd13972fb1ad/manager/0.log"
Oct 30 22:16:51.413120 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:51.413089    5886 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fx2nl/perf-node-gather-daemonset-5kmk4"
Oct 30 22:16:56.526036 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:56.526002    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fccbd9b79-fzlvq_71550ced-41dd-4cab-adf5-ac8cee060005/kube-auth-proxy/0.log"
Oct 30 22:16:56.658011 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:56.657968    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-cf7lb_f9457827-beb1-4755-bb94-29ac9c2c1659/istio-proxy/0.log"
Oct 30 22:16:56.662492 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:56.662466    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-cf7lb_f9457827-beb1-4755-bb94-29ac9c2c1659/istio-proxy/1.log"
Oct 30 22:16:56.958746 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:56.958705    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9x9cn_9120e132-28e6-49ac-a31e-d263a71cea20/payload-pre-processing/0.log"
Oct 30 22:16:57.023205 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:57.023176    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-6l9b4_ee881cfc-ad3c-4017-8504-722b0543c67f/payload-processing/0.log"
Oct 30 22:16:57.545914 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:57.545844    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zsrxr_6427a41a-4ada-4a89-a06a-b28b84815333/dns/0.log"
Oct 30 22:16:57.660960 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:57.660919    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zsrxr_6427a41a-4ada-4a89-a06a-b28b84815333/kube-rbac-proxy/0.log"
Oct 30 22:16:57.784481 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:57.784435    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fccbd9b79-fzlvq_71550ced-41dd-4cab-adf5-ac8cee060005/kube-auth-proxy/0.log"
Oct 30 22:16:57.919594 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:57.919559    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-cf7lb_f9457827-beb1-4755-bb94-29ac9c2c1659/istio-proxy/0.log"
Oct 30 22:16:57.922829 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:57.922802    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-cf7lb_f9457827-beb1-4755-bb94-29ac9c2c1659/istio-proxy/1.log"
Oct 30 22:16:58.222556 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:58.222425    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9x9cn_9120e132-28e6-49ac-a31e-d263a71cea20/payload-pre-processing/0.log"
Oct 30 22:16:58.243995 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:58.243959    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tsz8h_42109964-a1d9-479b-b626-76b60f9f49da/dns-node-resolver/0.log"
Oct 30 22:16:58.285586 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:58.285554    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-6l9b4_ee881cfc-ad3c-4017-8504-722b0543c67f/payload-processing/0.log"
Oct 30 22:16:59.734933 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:16:59.734901    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q5wbv_9193a5a4-d86a-47f5-9db2-c23851803a36/node-ca/0.log"
Oct 30 22:17:01.029900 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.029868    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fccbd9b79-fzlvq_71550ced-41dd-4cab-adf5-ac8cee060005/kube-auth-proxy/0.log"
Oct 30 22:17:01.163988 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.163955    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-cf7lb_f9457827-beb1-4755-bb94-29ac9c2c1659/istio-proxy/0.log"
Oct 30 22:17:01.167718 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.166542    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-cf7lb_f9457827-beb1-4755-bb94-29ac9c2c1659/istio-proxy/1.log"
Oct 30 22:17:01.331653 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.331578    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cc7x7_ab559120-5838-4b2d-9a35-c9c9d62c0dd7/manager/0.log"
Oct 30 22:17:01.389312 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.389284    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9snpv_e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83/kuadrant-console-plugin/0.log"
Oct 30 22:17:01.459661 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.459632    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9x9cn_9120e132-28e6-49ac-a31e-d263a71cea20/payload-pre-processing/0.log"
Oct 30 22:17:01.523510 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.523474    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-6l9b4_ee881cfc-ad3c-4017-8504-722b0543c67f/payload-processing/0.log"
Oct 30 22:17:01.557412 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.557379    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-q6j5d_914c9e20-a98b-4655-8405-7dfbfc030564/manager/0.log"
Oct 30 22:17:01.704392 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:01.704346    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q7jpk_6f413ea5-f172-43ed-bb8f-425675362ac7/manager/0.log"
Oct 30 22:17:02.232202 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:02.232151    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cc7x7_ab559120-5838-4b2d-9a35-c9c9d62c0dd7/manager/0.log"
Oct 30 22:17:02.295982 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:02.295952    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-9snpv_e8d9fcc9-ca3d-47f4-a3ed-fa04f5215c83/kuadrant-console-plugin/0.log"
Oct 30 22:17:02.521112 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:02.521023    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-q6j5d_914c9e20-a98b-4655-8405-7dfbfc030564/manager/0.log"
Oct 30 22:17:02.733871 ip-10-0-129-84 kubenswrapper[5886]: I1030 22:17:02.733834    5886 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q7jpk_6f413ea5-f172-43ed-bb8f-425675362ac7/manager/0.log"